SNS Message Partitioning for Large Reports

August 6, 2024

Introduction

The bearse-feature-reporting system has been enhanced with a new feature that automatically handles large reports by partitioning them into smaller chunks when they exceed the AWS SNS message size limit. This improvement ensures reliable delivery of large reports, particularly beneficial for asset reporting functionality.

Features

Automatic Message Partitioning

Implemented smart partitioning for messages exceeding 250KB (SNS limit is 256KB)
Each partition is individually compressed and transmitted
Maintains message integrity across partitions
Includes metadata for proper message reconstruction

Enhanced Message Handling

Improved compression logic for all messages
Added partition tracking and sequencing
Automatic account ID detection and timestamp assignment
Comprehensive logging for message sizes and partition status

Monitoring and Logging

Detailed logging of message sizes before and after compression
Partition progress tracking with size information
Clear visibility into the partitioning process

Examples

Partitioned Message Logging

The system now provides detailed logging for partitioned messages:

[INFO] 2024-07-31T02:34:03.721Z e4e7c5f2-3fc5-4692-9d0c-35e72be57774 Original message size: 805204 bytes, compressed: True
[INFO] 2024-07-31T02:34:03.765Z e4e7c5f2-3fc5-4692-9d0c-35e72be57774 Sending partition 1/4 with size: 251992 bytes
[INFO] 2024-07-31T02:34:04.062Z e4e7c5f2-3fc5-4692-9d0c-35e72be57774 Sending partition 2/4 with size: 252084 bytes
[INFO] 2024-07-31T02:34:04.266Z e4e7c5f2-3fc5-4692-9d0c-35e72be57774 Sending partition 3/4 with size: 251992 bytes
[INFO] 2024-07-31T02:34:04.471Z e4e7c5f2-3fc5-4692-9d0c-35e72be57774 Sending partition 4/4 with size: 55776 bytes
[INFO] 2024-07-31T02:34:04.506Z e4e7c5f2-3fc5-4692-9d0c-35e72be57774 report sent

Message Structure

Each partitioned message includes the following metadata:

Account ID
Timestamp
Total number of partitions
Current partition number
Compressed message content

Conclusion

This release significantly improves the reliability of the reporting system by automatically handling large reports through intelligent partitioning. The enhancement is particularly valuable for asset reporting where report sizes frequently exceed SNS limits. The system now handles these cases gracefully while maintaining data integrity and providing clear visibility into the process through comprehensive logging.

The feature is currently implemented for asset reporting but can be extended to other report types if needed in the future. Users of the reporting system will experience more reliable report delivery without any changes required on their end.