Advanced Configuration

JetS3t includes a number of configuration mechanisms that allow users with specialised requirements to control how the toolkit and applications will operate:

JetS3t Properties

The main configuration properties for the JetS3t toolkit and applications are stored in a file called jets3t.properties. This file specifies advanced communication properties. A properties file with default settings is included in the configs directory JetS3t release.

The jets3t.properties file is a standard text document that lists properties in the standard Java Properties file format, containing name value pairs and comments:

   # Comments start with a hash character.
   # Empty lines are ignored...

   # Properties are set as name=value, for example:
   propertyName=propertyValue
   my.favourite.colour=blue

Overview of jets3t.properties

Common S3Service communication properties
s3service.https-only If true, all communication with S3 will be via encrypted HTTPS connections, otherwise communications will be sent unencrypted via HTTP
Default: true
s3service.max-thread-count The maximum number of concurrent communication threads that will be started by the multi-threaded service for upload and download operations.
Note: This value should not exceed the maximum number of connections available, such as is set by the property httpclient.max-connections
Default: 4
s3service.admin-max-thread-count The maximum number of concurrent communication threads that will be started by the multi-threaded service for administrative operations such as DELETE and HEAD requests. Because these operations are short-lived and incur little bandwidth, most systems can reliably use many more administrative threads than upload/download threads.
Note: This value should not exceed the maximum number of connections available, such as is set by the property httpclient.max-connections
Default: 10
s3service.stream-retry-buffer-size How many bytes to buffer for use when retrying failed transmissions. This value must be small enough that applications using multiple upload threads will not exceed their available memory by buffering data
Note: Applies only to InputStream-based uploads, not uploads based on files
Default: 131072
s3service.internal-error-retry-max The maximum number of times each connection that fails with S3 InternalServer errors will be retried. To disable retries of InternalError failures set this to 0.
Note: After each failure the service waits before retrying. The time to wait is calculated with the formula:
50 * (internalErrorCount ^ 2)
Default: 5
s3service.default-bucket-location The location in which buckets will be created by default. This value may be US or EU.
Default: US
s3service.s3-endpoint The host name of the S3 service. You should only ever change this value from the default if you need to contact an alternative S3 endpoint for testing purposes.
Default: s3.amazonaws.com
s3service.disable-dns-buckets By default, JetS3t will always incorporate bucket names that are DNS-compatible into the host name of its requests. For example, a request addressing the bucket "mybucket" will be sent to the host name "mybucket.s3.amazonaws.com". If you set this property to true, JetS3t will put bucket names in the request's path instead of its host name, for example "s3.amazonaws.com/mybucket". This behaviour may be useful for testing purposes, or if DNS problems prevent the default host name addressing format from working correctly.
Note: If you set this property to true, you will be unable to access buckets located in the EU.
Default: false
s3service.ignore-exceptions-in-multi If this value is set to true, JetS3t will ignore exceptions that occur during multi-threaded operations performed by the S3ServiceMulti class. This option should only be used as a last resort when you need to complete an upload or download operation despite many communication errors.
Default: false
REST/HTTP HttpClient Implementation properties
httpclient.connection-timeout-ms How many milliseconds to wait before a connection times out. 0 means infinity.
Default: 60000
httpclient.socket-timeout-ms How many milliseconds to wait before a socket connection times out. 0 means infinity.
Default: 60000
httpclient.max-connections The maximum number of simultaneous connections to allow
Note: Be careful increasing this from the default value, as communications errors can result if it is set too high for your system/connection/usage scenarios
Default: 10
httpclient.stale-checking-enabled "Determines whether stale connection check is to be used. Disabling stale connection check may result in slight performance improvement at the risk of getting an I/O error when executing a request over a connection that has been closed at the server side."
Default: true
httpclient.retry-max How many times to retry connections when they fail with IO errors. Set this to 0 to disable retries.
Default: 5
httpclient.useragent The user agent string sent with HTTP requests.
Default: Application-specific
httpclient.read-throttle Imposes a rudimentary limit on the bandwidth used for uploads by throttling the speed at which data will be sent to S3. This property specifies the limit in KB/s, specified as an integer.
Default: N/A (This property is commented-out by default)
httpclient.proxy-autodetect Indicates that JetS3t should auto-detect the HTTP proxy settings appropriate for the local machine.
Default: true
httpclient.proxy-host Explicitly sets the host name of a HTTP proxy server. To apply this setting, proxy autodetection should be disabled by setting httpclient.proxy-autodetect to false.
Default: N/A (This property is commented-out by default)
httpclient.proxy-port Explicitly sets the port number of a HTTP proxy server. To apply this setting, proxy autodetection should be disabled by setting httpclient.proxy-autodetect to false.
Default: N/A (This property is commented-out by default)
httpclient.socket-receive-buffer Integer value for the TCP receive window size in bytes, which is provided as a hint to the kernel by the HttpClient library before HTTP socket connections are established. This value will not necessarily override a kernel's default TCP window size settings because the kernel is free to ignore the hint.
Default: N/A (This property is commented-out by default, in which case the default TCP window size will be used)
httpclient.socket-send-buffer Integer value for the TCP receive window size in bytes, which is provided as a hint to the kernel by the HttpClient library before HTTP socket connections are established. This value will not necessarily override a kernel's default TCP window size settings because the kernel is free to ignore the hint.
Default: N/A (This property is commented-out by default, in which case the default TCP window size will be used)
File Upload properties
uploads.storeEmptyDirectories Boolean value that indicates whether JetS3t applications that upload files to S3 should create place-holder objects to represent directories. Place-holder objects are named after a directory, have no data content and have the mimetype application/x-directory. If this option is set to false, synchronization from a local filesystem and S3 will not include empty directories.
Default: true
filecomparer.ignore-panic-dir-placeholders Boolean value that indicates whether JetS3t applications that synchronize files with S3 should ignore directory place-holder objects created by the Transmit application provided by Panic.
Default: N/A (This property is commented-out by default)
filecomparer.use-md5-files Boolean value that indicates whether JetS3t applications that synchronize files with S3 should look for files containing pre-generated MD5 hash values. These MD5 files must be named after the file they refer to, eg <filename>.md5, and must contain a hex-encoded MD5 value. If one of these files is present, JetS3t will use the pre-generated hash value rather than calculating a new hash - potentially saving a great deal of time when you are synchronizing large files. If the pre-generated MD5 file is older than the file it refers to, a new hash value will be calculated.
Default: N/A (This property is commented-out by default)
filecomparer.generate-md5-files Boolean value that indicates whether JetS3t applications that synchronize files with S3 should create <filename>.md5 files to store the MD5 hash values they calculate. If these values are stored, they need not be recalculated in future synchronizations. This property is intended to be used in conjunction with filecomparer.use-md5-files.
Default: N/A (This property is commented-out by default)
filecomparer.skip-upload-of-md5-files Boolean value that indicates whether JetS3t applications that synchronize files with S3 should avoid uploading <filename>.md5 files to S3. This property is intended to be used in conjunction with filecomparer.use-md5-files.
Default: N/A (This property is commented-out by default)
File Download properties
downloads.restoreLastModifiedDate Boolean value that indicates whether JetS3t applications that download files from S3 should restore the original last modified date of the file, according to the value of the metadata item "jets3t-original-file-date-iso8601" created by JetS3t tools that upload files. If this property is set to true, it will be possible to retain a file's last modified date when that file is downloaded or restored form S3.
Default: false
Encryption properties
crypto.algorithm Name of the PBE cryptographic algorithm to used by JetS3t applications that encrypt items uploaded to S3.
Note: The default algorithm isn't a strong one, and should be improved if security is important
Note 2: The algorithms available will depend on the cryptography provider libraries installed on a system, and the Java policy file settings. This is a complicated topic that is not specific to JetS3t, please refer to documentation elsewhere.
Default: PBEWithMD5AndDES
GUI properties
gui.verboseErrorDialog If true, JetS3t applications that display error dialogs will include more information about errors.
Default: true

Mimetypes

JetS3t applications, and any toolkit-based application that use the Mimetypes utility class, can recognise the mime/content type of a file based on the file's extension. For example, .html files are recognised as text/html. A properties file with default settings is included in the configs directory JetS3t release.

Mimetype recognition is configured in the mime.types configuration file, which is exactly the same format as is used by the Apache Web Server (in fact, we pinched the whole mimetypes file from Apache).

The mime.types file is a standard text document that lists the recognised mime types, each on a new line. The mimetype is listed, followed by one or more tabs, then one or more space-separated file extensions that map to the mimetype. The best way to understand the file format is to look at some examples:

text/html               html htm
text/plain              asc txt
application/pdf         pdf
audio/mpeg              mpga mp2 mp3

Ignore Files on Upload

JetS3t applications, and any toolkit-based application that uses the FileComparer utility class, will look for special ignore files named .jets3t-ignore when generating listings of local files to upload. These ignore files are plain text documents that contain file or directory names in the directory that should not be uploaded. Any file or directory that matches a name or a pattern in this file is completely ignored, as if it doesn't exist.
Users familiar with CVS will be familiar with this concept, as it copies the idea of .cvsignore files.

Ignore files are mainly useful when managing the backup of large file sets. For example, say you want to backup a folder called Documents but you don't want to backup a sub-folder in Documents called Private, nor do you want to backup all the .tmp files your word processor creates when you open a document. You could achieve this by creating a .jets3t-ignore file in the Documents folder with the content:

Private
*.tmp

The .jetset-ignore file format allows basic wild-card substitution:

  • Asterix (*): matches zero or more characters
  • Question mark (?): matches exactly one character

Note that the contents of the ignore file only apply to the immediate directory containing the .jets3t-ignore file. For example, the wildcard string *.tmp will only ignore .tmp files in the current directory, any temporary files in subdirectories will still be included. Similarly, folder paths cannot be used - the following statements will not work:

Private/*.tmp
Subfolder/PrivateFolder