Scaling Django Applications with S3-Synced SQLite

Scaling Django Applications with S3-Synced SQLite

Are you looking for a cost-effective way to scale your Django applications? Look no further! In this article, we will explore how to use S3-Synced SQLite as a backend database engine to optimize the scalability and performance of your Django applications.

The Power of S3-Synced SQLite

S3-Synced SQLite is a game-changer for high-read applications that don’t have concurrent writes, such as a CMS for your blog. By leveraging the power of AWS S3 and SQLite, you can significantly reduce costs compared to traditional database solutions like AWS RDS or Aurora.

However, it’s important to note that concurrent writes can be problematic with S3-Synced SQLite. Writes may not show up in concurrent readers, as the database is transferred between S3 storage and the Lambda instance for each request. This trade-off is crucial to consider when deciding to adopt this solution.

Getting Started

To start using S3-Synced SQLite, you’ll need to install the django-s3-sqlite package via pip:

#
$ pip install django-s3-sqlite

Next, add the package to your Django installed apps:

#python
INSTALLED_APPS += ["django_s3_sqlite"]

Configuring Django Settings

The next step is to configure the Django settings to use the S3-Synced SQLite database engine. Update your DATABASES settings as follows:

#python
DATABASES = {
    "default": {
        "ENGINE": "django_s3_sqlite",
        "NAME": "sqlite.db",
        "BUCKET": "your-db-bucket",
        "AWS_S3_ACCESS_KEY": "YOUR_ACCESS_KEY",  # optional, for S3 bucket lockdown
        "AWS_S3_ACCESS_SECRET": "YOUR_ACCESS_SECRET",  # optional, for S3 bucket lockdown
    }
}

Note that you can optionally lock down your S3 bucket to an IAM user by providing the access key and secret in the settings.

SQLite Version Compatibility

It’s crucial to ensure SQLite compatibility with your Django version. Newer versions of Django (v2.1+) require a newer version of SQLite (3.8.3+) than what is available on AWS Lambda instances (3.7.17).

To overcome this, download the _sqlite3.so file for your Python version from the shared-objects directory of the django-s3-sqlite repository. Place the file at the root of your Django project. Additionally, add the following line to your Zappa JSON settings file for each environment:

#
"use_precompiled_packages": false

These additional steps are necessary to ensure proper SQLite compatibility in a serverless environment.

Database Maintenance

Since SQLite keeps the database as a single file, it’s essential to keep it small and defragmented. Regularly perform a database vacuum, especially after deleting or updating data. You can use the following command to vacuum your database:

#bash
zappa manage [instance] s3_sqlite_vacuum

Creating a Default Admin User

To manage your application effectively, you’ll likely need a default admin user. You can create one using the following command:

#
$ zappa manage  create_admin_user

You can also pass arguments to customize the admin user creation:

#
$ zappa manage  create_admin_user one two three

Internally, this command calls User.objects.create_superuser('one', 'two', 'three').

Conclusion

Scaling Django applications has never been easier and more cost-effective with S3-Synced SQLite. By following the steps outlined in this article, you can optimize the scalability and performance of your applications while keeping costs low.

Remember to carefully consider the trade-offs of S3-Synced SQLite, particularly the limitations with concurrent writes. If your application relies heavily on concurrent writes, alternative solutions may be more suitable.

If you have any questions or need further assistance, don’t hesitate to reach out. Happy scaling!

References

• S3-Synced SQLite – A Serverless Relational Database
• django-s3-sqlite on PyPI
• Building _sqlite3.so
• FlipperPA/django-s3-sqlite on GitHub

Acknowledgements

• Special thanks to Rich Jones for the inspiration and initial work on this project.
• Thanks to Tim Allen and Peter Baumgartner for maintaining the project.