------------- Version 0.8.0 ------------- TOOLKIT * General - New generic bucket, object, and multi-threaded service classes for interacting with either of the S3 or Google Storage services. - No long compatible with JDK 1.4. JDK 6+ is now required to run JetS3t. - Removed antiquated and unsupported SOAP S3 service implementation from toolkit. - FileComparer and related tools are now compatible with both S3 and Google Storage services. - FileComparer tool, and therefore all JetS3t apps, now uses a trailing slash to denote directory placeholder objects in S3 instead of the custom "application/x-directory" content-type. This should improve compatibility with other storage management tools, especially Amazon's AWS Console. * JetS3t Property File changes - Renamed "s3service.internal-error-retry-max" to "storage-service.internal-error-retry-max" - Renamed "s3service.stream-retry-buffer-size" to "uploads.stream-retry-buffer-size" - Renamed "s3service.defaultStorageClass" to "s3service.default-storage-class" - Added properties for new ThreadedStorageService: "threaded-service.max-thread-count" "threaded-service.admin-max-thread-count" "threaded-service.ignore-exceptions-in-multi" * ThreadedStorageService / SimpleThreadedStorageService - New multi-threaded service classes that allow operations to be performed in bulk on either the RestS3Service or GoogleStorageService service implementations. * GoogleStorageService - Google Storage (GS) service class added to provide specific support for Google's storage provider. * S3Service - Added support for bucket policy documents: set, get, and delete. - Methods for setting and using AWS DevPay credentials moved from S3Service to RestS3Service. * CloudFront Service - Added support for setting Default Root Object for CloudFront distributions. - Added support for API version 2010-08-01 and invalidation of objects. - Modified CloudFrontService#signUrl and #signUrlCanned to accept raw resource URL or path strings, instead of expressing all resources as HTTP URLs. This allows for URL signing of HTTPS and RTMP streaming resources. * SOAP implementation - REMOVED from project. COCKPIT - Added explicit support for logging in to either the Google Storage or Amazon S3 service in the startup/login dialog. - A CloudFront distribution's Default Root Object setting can be configured in the "CloudFront Distributions" dialog. SYNCHRONIZE - Directly supports new GoogleStorageService, allowing synchronization with Google Storage accounts through the native API. - Choose target service end-point with the new "--provider" command argument. KUDOS TO: - Claudio Cherubino of Google Inc., for significant refactoring work and the addition of full support for the Google Storage service. - David Kocher for a patch to implement Default Root Object support in the CloundFrontService, and for his invaluable testing and feedback. ------------- Version 0.7.4 ------------- TOOLKIT * General - Added support for the new Reduced Redundancy Storage (RRS) class for objects. This discounted storage class can be applied when an object is created, overwritten or copied. The storage class can be set per-object, or a default value can be set with the property 's3service.defaultStorageClass'. Storage classes can be disabled altogether for services that don't support them using the 's3service.enable-storage-classes' property. - Added support for buckets located in the Asia Pacific (Singapore) location "ap-southeast-1". - Eucalyptus-friendly generated URLs (#14). JetS3t configuration properties 's3service.s3-endpoint-virtual-path', 's3service.s3-endpoint-http-port' and 's3service.s3-endpoint-https-port' are no longer ignored when generating signed or torrent URLs. - Bash shell scripts now support cygwin by automatically converting classpaths to Windows format. - Improved configurability and flexibility by removing dependence on the VM-wide static Constants#S3_HOSTNAME constant value (see API Changes below) * CloudFront Service - NOTE: These changes break backward-compatibility for some of the CloudFrontService and related class interfaces. - Updated service to support API version 2010-06-01 which adds support for HTTPS-only distributions (a.k.a. RequiredProtocols) - Updated service to support API version 2010-05-01 which adds logging support for streaming distributions. * API Changes - Removed support for methods and metadata attributes that have been obsolete (and deprecated) since version 0.6.0. - Deprecated static methods in S3Service for generating signed URLs. The new non-static method equivalents should be used from now on to avoid dependency on a VM-wide S3 endpoint constant. * SOAP implementation - Deprecated in preparation for imminent removal from project. COCKPIT - Added support for the REDUCED_REDUNDANCY storage class to the Preferences dialog (set default storage class for uploads), the Copy or Move dialog (set the storage class for destination objects), and the Object Attributes dialog (see the current storage class property of objects) - Added support for buckets located in the Asia Pacific (Singapore) location. - Added support for creating HTTPS-only CloudFront distributions. - Removed overly fussy string-length checks in credentials dialog. SYNCHRONIZE - Allow synchronization with third-party buckets that are not owned by the user, provided the user has the necessary access to that bucket. KUDOS TO: - Alexey I. Froloff for Eucalyptus-friendly generated URLs patch. - David Kocher for patches to allow advanced users to provide their own RequestEntity classes when uploading objects, and to improve HTTP retry handling. ------------- Version 0.7.3 ------------- TOOLKIT * General - Added support for the new S3 Bucket Versioning feature, and for Multi-Factor authenticated deletes. - S3ServiceMulti#downloadObjects method now automatically verifies that the data received matches the ETag MD5 hash value provided by S3. JetS3t's applications, and any third-party clients that use this method, will therefore get automatic data verificaiton of downloads similar to that provided (in most cases) for uploads. - Removed methods incompatible with JDK version 1.4. * CloudFront Service - Updated service to support API version 2010-03-01 which allows streaming distributions to be made private. - Fixed bug parsing that prevented AWS Account Numbers from being parsed from XML responses into a DistributionConfig object. COCKPIT - Fixed bug that caused Cockpit to create buckets in the US Standard location regardless of the location selected by the user. SYNCHRONIZE - When an encryption password or credentials file password must be entered by a user at the command prompt the characters are no longer echoed to standard output when run on JDK 1.6 or greater. KUDOS TO: Bennett Hiles for echo-less password input for Synchronize and removal of methods incompatible with JDK 1.4. ------------- Version 0.7.2 ------------- TOOLKIT * General - Added support for private and streaming CloudFront distributions (see the "CloudFront Service" section below) - Changed default thread and connection count settings to improve reliability when tranferring many large files and performance when performing object administration tasks. Default property settings for maximum simultaneous admin threads and HTTP connections are increased from 10 to 20, while the default number of non-admin threads is reduced from 4 to 2. - Improved compatibility with other S3 tools by setting the following properties to true by default: "filecomparer.ignore-panic-dir-placeholders" "filecomparer.ignore-s3organizer-dir-placeholders" - Added new property "filecomparer.ignore-s3organizer-dir-placeholders" which allows JetS3t to ignore directory place-holder objects created by the S3 Organizer Firefox extension when uploading or downloading files. - Added a new property "filecomparer.md5-files-root-dir" which allows MD5 files generated for file comparison purposes to be stored in a custom path, instead of within the same directory structure as data files. - Added support for the property "filecomparer.skip-symlinks" which causes symlink/alias files to be ignored by JetS3t applications when deciding which files to send to S3. Although "soft" links can be detected and ignored, hard links cannot be detected and will be treated as ordinary files. - Added support for the TargetGrants setting of logged S3 buckets. This allows you to set default ACL access permissions for S3 bucket logs generated by the service. - JetS3t's JMX instrumentation delegate is now re-initialized each time a new S3Service is created, allowing users to enable/disable JMX logging by adjusting System properties after a JVM has started. Note that it is still not possible to enable/disable Bucket- or Object-specific event logging after the JVM has started. - JMX logging for JetS3t can be enabled on Java 1.6 systems by including the "jets3t.mx" system property, since the "com.sun.management.jmxremote" property is no longer required to turn on JMX support in the JVM. - Improved the Windows batch scripts to correctly detect the JETS3T_HOME location regardless of the current directory when the JetS3t script is run. * CloudFront Service - API support for creating and managing private distributions: - create and manage Origin Access Identifiers as required for private distributions. - generate canned and custom-policy signed URLs for private distributions protected with mandatory request signing. - API support for creating and managing Streaming Distributions. * REST Implementation - Added sanity-checking of HTTP header values as obtained from an S3Object's metadata map. Non-ASCII header names and header duplicates caused by capitalization mismatches now fail early with descriptive S3ServiceException errors. - Fixed bug in signed-url based uploading that caused automatic ETag verification to fail with a NullPointerException if an ETag value wasn't set prior to upload. - Fixed NullPointerException in error reporting when REST service is run without a network available. - Added property setting "httpclient.connection-manager-timeout" which allows users to specify a timeout when an S3 operation is waiting for a connection to become available in the HttpClient connection pool. The default setting of 0 means wait indefinitely. SYNCHRONIZE - You can now set custom metadata information when uploading objects to S3 using Synchronize property names prefixed with "upload.metadata.", for example: upload.metadata.Cache-Control=max-age=300 upload.metadata.Expires=Thu, 01 Dec 1994 16:00:00 GMT upload.metadata.my-metadata-item=This is the value for my metadata item - Synchronize can be told to continue even if local files or folders included in the UPLOAD command are missing or unreadable, by setting the "upload.ignoreMissingPaths" property to true. Normally Synchronize would consider this an error and halt. WARNING: This property could be dangerous if used unwisely because it could cause legitimate objects in S3 to be deleted due to problems on the local file system. Beware! KUDOS TO: Christian Mallwitz for informing us of the %~dp0 script variable that greatly improves the Windows batch scripts. Jawahar Lal Nayak for: - a patch that contributed to the new TargetGrants support for logged S3 buckets. - bug report about the NPE error condition when the REST implementation is run on a machine without a network connection available. Benjamin Schmaus for patches to configure the REST connection pool timeout, and to make HTTP response headers available for failed REST connections via S3ServiceExceptions. Justin C for a patch that inspired Synchonize's ability to upload files with custom metadata using "upload.metadata." properties. ------------- Version 0.7.1 ------------- COCKPIT - Fixed a menu display bug that caused the bucket and object action menus to appear behind other GUI elements on some Windows systems. - The Logging Status settings of CloudFront distributions can now be managed within the CloudFront Distributions dialog box (see the Tools menu). - New "Switch login" Service menu item allows uses with multiple S3 accounts to quickly switch between them, once the initial log in has been performed. TOOLKIT * General - Added support for JMX instrumentation of S3Service functions. To enable JMX set the system property "com.sun.management.jmxremote". Instrumentation for S3Bucket and S3Object MBeans is disabled by default, to enable this reporting set the following system properties to "true": "jets3t.bucket.mx", "jets3t.object.mx" - Added simplified constructors for S3Object, so the object's bucket need not be specified in advance. This is most useful when uploading data. - Added method getAccountOwner() to S3Service which allows you to look up the ID and display name of an S3 account's owner, even if the account contains no buckets. - Tweaks to improve support for using JetS3t with the open source Eucalyptus cloud computing service. * REST Implementation - Fixed a configuration error that caused the REST implementation to be limited to 20 simultaneous connections, regardless of the "httpclient.max-connections" property setting. Now the JetS3t property "httpclient.max-connections" sets the global connection limit, while the optional property "httpclient.max-connections-per-host" sets the per-host connection limit. * CloudFront Service Implementation - Added support for the new Logging Status feature that allows log files for activity in your CloudFront distributions to be sent to an S3 bucket. KUDOS TO: Laxmilal Menaria of Chambal.com Inc. for a patch that contributed to the new S3Service#getAccountOwner feature. Doug MacEachern of Hyperic.com for contributing the bulk of a JMX instrumentation implementation. peter_griess, askwar and leebutts (java.net users) for identifying bugs and providing patches or solutions. ------------- Version 0.7.0 ------------- COCKPIT - Added ability to view and manage Amazon CloudFront distributions using a dialog box. - Added support for configuring Requester Pays buckets. - Improved support for DevPay credentials, which can now be set and saved directly in the login dialog box instead of only in a properties file. - User preferences can now be remembered on your computer, so they do not need to be re-set every time you start Cockpit. - Improved dialog box for generating Signed URLs. URLs for multiple objects can now be generated at once. SYNCHRONIZE - AWS credentials and the cryptographic password can now be provided via prompts on the command-line, rather than merely through a properties file. - Added the --credentials option, which allows AWS credentials to be loaded from an encrypted file rather than an insecure properties file. The encrypted file can be created with the AWSCredentials API or the Cockpit application. - Synchronize will act as an anonymous client if empty values are provided as the AWS access and secret keys. When the S3 connection is anonymous, only public buckets will be accessible. - Fixed a bug that prevented Synchronize from recognizing duplicate file names when uploading to an S3 subdirectory path when using the --batch option. - Synchronize will now prompt for HTTP Proxy login credentials if required. If proxy credentials are pre-specified in the jets3t.properties file, you should not be prompted. - Improved handling of uploads of many files where the files must first be transformed (compressed or encrypted). In this case, files are now transformed and uploaded in batches, where the batch size is set by the synchronize.properties setting 'upload.transformed-files-batch-size' TOOLKIT * General - Added initial implementation of CloudFront service. - Added property "cloudfront-service.internal-error-retry-max" for defining the retry limit for CloudFront internal service errors. - Added support for configuring and accessing Requester Pays buckets. - Added property "httpclient.requester-pays-buckets-enabled" for defining whether the RestS3Service supports Requester Pays buckets by default. - Improved support for accessing S3 using DevPay credentials with a new credentials class for defining and storing these credentials: AWSDevPayCredentials - Added a class (AWSDevPayProduct) to represent a DevPay product, and to load information about pre-defined products from the properties file devpay_products.properties. - Improved the interpretation of .jets3t-ignore files so that child paths or arbitrary paths can be specified and ignored. For example, to ignore all CVS directories you can now add a single ignore path at the top level: **/CVS * REST Implementation - Added support for proxy authentication settings in jets3t.properties: "proxy-user", "proxy-password", and "proxy-domain" - The service's HTTP proxy settings can be updated an reapplied on-demand using a range of #initHttpProxy methods. - Added property settings to allow the default HTTP and HTTPS ports to be changed, which can be handy when running the service through a proxy for testing: s3service.s3-endpoint-http-port, s3service.s3-endpoint-https-port - The HttpConnectionManager and HttpClient objects used by the REST service can now be reinitialised on-demand using the #initHttpConnection method. - The underlying HttpClient and HttpConnectionManager objects can be accessed to provide greater control and flexibility after a service is created. - The automatic time adjustment performed in response to RequestTimeTooSkewed errors will now work through proxies. KUDOS TO: David Kavanagh for sample code that helped improve HTTP proxy support and configurability. Nikolas Coukouma of Zmanda Inc. for patches to significantly improve support for Amazon DevPay in the library and Cockpit applications. Allan Frank for a patch that helped improved the Synchronize application's handling of uploads. ------------- Version 0.6.1 ------------- COCKPIT - Added support for copying or moving objects within and between buckets. - Added dialog to assist in moving/renaming multiple objects within a bucket. - Added support for updating objects with new metadata values. - Fixed issue where downloading items to a directory containing many files and directories (eg 250K+) would cause Cockpit to fail with OutOfMemory errors. - Added a Confirm Password field to the encryption settings, to help prevent problems where a password is mistyped. - Cockpit can now generate valid signed GET URLs for DevPay S3 accounts. SYNCHRONIZE - Synchonize now recognizes when you have specified a partial S3 path when synchronizing DOWN. A partial path is one that does not exactly match a directory path. Partial paths can now act as a prefix test that identifies objects in an S3 path, where only the objects in S3 that match the prefix are downloaded. For example, in a bucket that has a Docs subdirectory containing objects named with a timestamp prefix, the S3 path "my-bucket/Docs/2008" will identify the objects in the Docs subdirectory with names beginning with 2008. - Added --batch option that causes Synchronize to compare and download/upload files in batches of 1000, rather than all at once. This option will reduce the memory required to synchronize buckets with many objects, and will allow file transfers to commence as soon as possible rather than after the slow comparison process. - Added --skipmetadata option that causes Synchronize to skip the retrieval of object metadata information from S3. This makes synchs much faster for large buckets, but leave the app with less info to make decisions. - Added --reportlevel option that allows the user to control how much report detail is printed, from 0 (no reporting) to 3 (all reporting). - Added --move option that deletes local files after they have been uploaded to S3, or deletes objects from S3 after they have been downloaded. - Property settings in the synchronize.properties file, or in the file referred to by the --properties option, will override properties of the same name in jets3t.properties. This makes it easy to create task-specific properties files. TOOLKIT * General - Added support for the new Copy Object functionality that allows you to copy, move, rename and update objects in your S3 account. These operations are available through the S3Service as the methods copyObject, moveObject, renameObject and updateObjectMetadata. - Added support for conditional copying of objects, based on ETag and Modified Date comparisons. - Made it easier to verify that your data is correctly stored in S3 without corruption during transit, by calculating Content-MD5 hash values by default in commonly-used S3Object constructors and providing S3Object#verifyData methods that make it easy to verify data downloaded from S3. - Added basic support for accessing DevPay S3 accounts. DevPay user and product tokens can be provided to the S3Service class directly using the #setDevPayUserToken and #setDevPayProductToken methods, and default token values can be specified in the jets3t.properties file using the settings "devpay.user-token" and "devpay.product-token". - Modified Bucket Logging Status changing behaviour to update ACL settings of the target bucket before (re)directing log files to the bucket. - Fixed bug in RestS3Service that caused failures in some circumstances when buckets were created with the "US" location. - S3Service instances can now be configured individually by providing a JetS3tProperties object when constructing the service. The property values in this object can also be updated programmatically after the object has been constructed. - FileComparer now supports an optional setting that makes it assume local files are the latest version when there is a clash between the modification dates and hash values of a local file and an object stored in S3. This option should only be used as a work-around for users who synchronize Microsoft Excel files to S3, as these are the only documents that exhibit the mismatch. To enable the option, set the jets3t.properties item "filecomparer.assume-local-latest-in-mismatch" to true. - Added support to FileComparer for listing objects based on prefix "partitions" that allow a bucket to be listed by multiple simultaneous threads. This can speed up listings for large buckets containing many virtual subdirectories. To use multiple partitions when building S3 object maps, configure the jets3t.properties item "filecomparer.bucket-listing." to specify a delimiter string and a traversal depth value separated by a comma, eg: filecomparer.bucket-listing.my-bucket=/,2 to partition the bucket my-bucket based on the '/' delimiter to a depth of 2 virtual subdirectories. * REST Implementation - Added support for the new Copy Object functionality, and conditional copying. - The XML parsing code now copes better when object names contain a carriage return character. A work-around is applied to prevent the Java XML parser from misinterpreting these characters, a fault which could cause some objects to become un-deletable. This new feature is enabled by default, but can be disabled with the jets3t.properties setting "xmlparser.sanitize-listings". - Changed default AuthenticationPreemptive setting for HttpClient connections from true to false, to improve compatibility with NTLM proxies. Preemptive auth can be turned on by setting "httpclient.authentication-preemptive" to true in jets3t.properties. - Refactored repeatable input streams implementations to recognize and support standard InputStream objects that can be reset. Thanks to Keith Bonawitz for the idea and patch for this. * Multi-threaded Service - Added a new listObjects method that performs a multi-threaded listing of a bucket's contents. You provide an array of prefix strings which serve to divide your objects into a number of "partitions", and the service performs these prefix-based listings in parallel. See src/org/jets3t/samples/ThreadedObjectListing.java for example usage. - Added a new notification event ServiceEvent#EVENT_IGNORED_ERRORS that provides information about exceptions that were ignored during a multi-threaded S3 operation. Exceptions are only ignored if the JetS3t property "s3service.ignore-exceptions-in-multi" is set to true. KUDOS TO: Shlomo Swidler for numerous improvements to the project's technical documentation and code. Keith Bonawitz for a patch to improve the handling of data re-transmissions using reset-able input streams. ------------- Version 0.6.0 ------------- COCKPIT - Added support for buckets located in Europe (EU location) - Generate signed GET URLs based on virtual host names. - Encryption algorithm can be changed in the preferences dialog. - Progress dialogs modified to fix display problems with JDK version 6. - Fixed bug causing encrypted files not to be automatically recognized and decrypted in some cases SYNCHRONIZE - Added --properties option that allows an explicit properties filename to be specified on the command line. - Added --nodelete option that keeps items on the destination that have been removed from the source. This is similar to --keepfiles except that files may still be reverted with the --nodelete option. - Added --noprogress option that prevents progress messages from being printed to the console, for cases where these messages pollute log files. This option is similar to --quiet, except the action report is printed. - Added the --acl option that allows Access Control List settings to be applied on the command line. Value must be one of: PRIVATE, PUBLIC_READ, PUBLIC_READ_WRITE - Added property setting to ignore missing source directories when uploading. - Progress status messages shortened to fit into 80 character width consoles. UPLOADER - Improved skinning capabilities. COCKPIT LITE - New Gatekeeper-compatible client application that allows almost all S3 operations to be performed by a client via Gatekeeper-provided signed URLs. GATEKEEPER - Extended interfaces and default implementations to provide URL signing for all operations performed by the new Cockpit Lite application. - Improved support for web proxies. TOOLKIT * General - S3Service class can now generate POST forms to allow web browsers to upload files or data directly to S3. - Added support for buckets located in the EU. Due to this change, the "s3service.end-point-host" property is now obsolete. - Added a default bucket location configuration setting in jets3t.properties: "s3service.default-bucket-location". If you do not use an explicit bucket location when you use the bucket creation API method, the bucket will be created in this default location. - Fixed bugs that caused uploads greater than 2GB to fail (Expect: 100-Continue is now supported) - Added the Bouncy Castle security provider to the suite, increasing the number of cipher algorithms available for encrypting files. - The method S3Service.listObjectsChunked now includes Common Prefixes in the result object returned. - Corrected the metadata name for the original date object metadata item, which was originally mis-typed as "jets3t-original-file-date-iso860". This name is now "jets3t-original-file-date-iso8601" (added "1" to the end). JetS3t tools remain compatible with the original metadata name. - The FileComparer class used by JetS3t applications can now be configured to use pre-generated .md5 files as a source for MD5 hash values, removing the need for hash values to be calculated for every synchronization operation - and potentially saving a great deal of time when you are synchronizing large files whose content changes rarely. This feature is controlled by a group of propery settings in jets3t.properties: filecomparer.use-md5-files, filecomparer.generate-md5-files, filecomparer.skip-upload-of-md5-files. - File listings as built using the FileComparer.buildFileMap() methods can be set not to include place-holder objects to represent empty directories (mimetype "application/x-directory") with the new configuration setting uploads.storeEmptyDirectories. * REST S3 Service - Added support for configuring TCP window sizes. In jets3t.properties, the settings httpclient.socket-receive-buffer and httpclient.socket-send-buffer are applied to the Socket send and receive buffers used by the underlying HttpClient library. - The REST implementation can now automatically cope with RequestTimeTooSkewed errors caused by the client computer's clock disagreeing with S3 about the current time. If this happens, JetS3t will look up the time according to S3 and compensate for the difference between the S3 clock and the client's clock. - Rudimentary upload bandwidth throttling using the jets3t.properties setting httpclient.read-throttle, which is specified in KB/s. - Proxy settings are now configurable via the jets3t.properties settings: httpclient.proxy-autodetect, httpclient.proxy-host, httpclient.proxy-port - Upgraded HTTPClient library to version 3.1 * Multi-threaded Service - Administration actions can have more threads assigned than upload/download actions with the new configuration setting s3service.admin-max-thread-count. These connections run much faster with more threads and as they are light weight they less likely to fail with higher thread counts. - Fixed S3ServiceSimpleMulti class to work with download sets larger than the number of available HTTP connections. Objects' data is now cached in temp files automatically. - The S3ServiceMulti@downloadObjects methods will restore the original last modified date of downloaded objects if the downloads.restoreLastModifiedDate configuration property is set to true. As these methods are used by the JetS3t applications, setting this property to true will allow file dates to be retained across synchronizations performed by Cockpit and Synchronize. The original last modified date must be available in the object's metadata item named "jets3t-original-file-date-iso8601". KUDOS TO: Alexis Agahi for a fix to BytesProgressWatcher. Pradyumna Lenka for an example update to support the EU bucket location. Andrea Barbieri of Moving Image Research for suggestions, feedback and quality control. SPONSORS: The JetS3t project has been generously supported with sponsorship from Moving Image Research : http://www.movingimageresearch.com/ ------------- Version 0.5.0 ------------- COCKPIT - Login credentials can now be stored in S3. - Added a dialog for configuring Bucket Logging. - Objects can be filtered by prefix and/or delimiter strings. - Third-party buckets can be added without first logging in. - User can cancel the listing of objects in large buckets. - Files being sent to s3 are only opened when necessary, removing the potential for too many files to be open at once (exceeding an Operating System imposed limit). - When uploading files, specific file/directory paths can be ignored using .jets3t-ignore settings files. - Access Control List permissions of uploaded files can be set to PRIVATE, PUBLIC_READ or PUBLIC_READ_WRITE SYNCHRONIZE - Progress status messages are displayed for long-running processes. - Files being sent to s3 are only opened when necessary, removing the potential for too many files to be open at once (exceeding an Operating System imposed limit). - When uploading files, specific file/directory paths can be ignored using .jets3t-ignore settings files. - Access Control List permissions of uploaded files can be set to PRIVATE, PUBLIC_READ or PUBLIC_READ_WRITE UPLOADER A new applet/application to allow third parties without AWS accounts or credentials to upload files to an S3 account. The Uploader provides a simple wizard-based GUI allowing end-users to provide information, choose the file(s) they will upload, and see upload progress. The Uploader is highly configurable via the uploader.properties file, with settings to configure: user input fields, explanatory text messages, names/images/tooltips for buttons at each stage in the wizard, basic skinning of the Uploader (an example HTML-like skin is included), and branding. The Uploader is designed to work closely with a Gatekeeper server, which provides the Uploader with signed URLs to allow it to perform uploads. GATEKEEPER A new servlet that acts as a Gatekeeper server for S3 operations. The servlet receives requests for S3 operations (GET, HEAD, PUT, DELETE) and responds to these requests with either a signed URL allowing the operation, or a message stating that the operation will not be allowed. Basic decision-making functionality is included with the Gatekeeper Servlet, however it is straight-forward to obtain more advanced control over the Gatekeeper's behaviour by implementing the relevant Java interfaces. There are specific interfaces for: allowing/denying requests, signing URLs, and assigning unique transaction IDs for a request. The Gatekeeper is designed to work closely with the Uploader application, providing it with signed URLs so the Uploader can add items to an S3 account without the end-user having any access to the AWS account credentials. TOOLKIT * General - Properties for many aspects of jets3t behaviour can be set by the user in a properties file. - Support for getting/setting Server Access Logging for buckets. - Improved encryption mechanism, which now uses PBE-based encryption and allows users to set their preferred algorithm. NOTE: All changes *should* be backwards compatible, but due to these changes there may be an increased risk of data loss for encrypted items. - New methods to chunk bucket listings, allowing for better handling of buckets with many objects. - A limit to the maximum number of simultaneous communication threads (ie those interacting with S3) is now imposed. This limit can be set in the properties file, and defaults to 50. - Signed URLs can be generated for GET, HEAD, PUT and DELETE requests - Fixed bug where object keys/names with special characters were not correctly encoded. - DNS caching is limited to 300 seconds. - When an object's data comes from a file, the file can be opened only when necessary rather than being opened as soon as the object is created. - Added documentation for advanced options settings and logging * REST/HTTP implementation - Requests that fail due to S3 Internal Server error are retried a configurable number of times, with an increasing delay between each retry attempt. - The REST/HTTP implementation is now less fussy about object key names, and will allow unusual names such as full URL strings etc. - Can detect (in some circumstances) a browser's proxy settings when run inside an applet context, and allows for callbacks to a credentials provider object when authentication is required (eg for proxies requiring username/password) - Signed URLs can be used to perform GET, HEAD, PUT and DELETE operations without the need for knowledge of AWS credentials. - Added a utility method putObjectWithSignedUrl to upload objects to S3 using only a signed PUT URL (ie no AWS credentials are required). - Configurable user agent string. - Sends an upload object's MD5 data hash to S3 in the header Content-MD5, to confirm no data corruption has taken place on the wire. * SOAP implementation - Tests for data corruption on the wire by matching ETag returned by S3 with expected MD5 hash value. * Multi-threaded Service - Signed URLs can be used to perform GET, HEAD, PUT and DELETE operations of multiple items at a time, without the need for knowledge of AWS credentials. KNOWN ISSUES * General - Uploading or downloading multiple large files can result in connection errors if the network connection is flooded. The chances of such errors can be reduced by using low values (eg 2) for for the jets3t.properties settings s3service.max-thread-count and httpclient.max-connections. * Cockpit - Copy & paste and Drag & drop doesn't work on some versions of Linux, making it difficult to enter AWS credentials. SPONSORS The JetS3t project has been generously supported with sponsorship from the following organisations. * Moving Image Research : http://www.movingimageresearch.com/ Moving Image Research (MIR) is a technology company with deep roots in media science offering software, network services, and consulting. CONTRIBUTORS Thankyou to the following contributors, who helped make this release possible: - Moving Image Research, Andrea Barbieri (http://www.movingimageresearch.com/) - Angel Vera (gunfus) ------------- Version 0.4.0 ------------- Initial public release.