Checking uploads¶
An app where users upload an avatar and documents, then download them again. Every file is
judged by its bytes, not by its name or the Content-Type the client sent:
FastAPIUploads checks avatars (PNG, JPEG, GIF or WebP, at most
2 MiB and 16 million pixels) as a dependency, and documents (PDF, Word, Excel and PowerPoint,
up to five at a time) with check(). Files are stored under names of the app's own and served
back with the type found in them.
BodySizeLimitMiddleware gives each route a body
limit, and SecurityHeadersMiddleware adds
X-Content-Type-Options: nosniff to every response, so browsers don't second-guess the type.
Run it¶
From a clone of the repository:
make deps # .venv, with every extra
source .venv/bin/activate
uvicorn examples.uploads.main:app
Files go to a temporary directory, deleted at shutdown; set UPLOAD_DIR to keep them.
Try it¶
An image is stored, with the type and dimensions read from its bytes:
$ curl -i http://127.0.0.1:8000/avatars -F file=@cat.png
HTTP/1.1 201 Created
content-type: application/json
x-content-type-options: nosniff
{"id":"6b88b761-8ff4-4477-bc9e-568420bd4205","name":"cat.png","content_type":"image/png","size":179,"width":64,"height":64}
An HTML page sent as cat.png, with Content-Type: image/png, is refused. Served back as it
was declared, it would run its script in the browser:
$ printf '<!doctype html><script>alert(document.cookie)</script>' > cat.html
$ curl -i http://127.0.0.1:8000/avatars -F 'file=@cat.html;filename=cat.png;type=image/png'
HTTP/1.1 415 Unsupported Media Type
{"detail":"Unsupported file type"}
A PNG of 33 bytes whose header says 50,000 × 50,000 pixels would take gigabytes of memory to decode. Its dimensions are read from the header, without decoding it:
$ python -c "import struct, sys, zlib; h = b'IHDR' + struct.pack('>IIBBBBB', 50000, 50000, 8, 2, 0, 0, 0); sys.stdout.buffer.write(b'\x89PNG\r\n\x1a\n' + struct.pack('>I', 13) + h + struct.pack('>I', zlib.crc32(h)))" > bomb.png
$ curl -i http://127.0.0.1:8000/avatars -F file=@bomb.png
HTTP/1.1 413 Content Too Large
{"detail":"Image too large"}
A body over the route's limit is refused from its Content-Length, before the multipart parser
stores it:
$ head -c 3145728 /dev/urandom > big.jpg
$ curl -i http://127.0.0.1:8000/avatars -F file=@big.jpg
HTTP/1.1 413 Content Too Large
x-content-type-options: nosniff
{"detail":"Request body too large"}
Several documents at once. The second is a Word document named minutes.pdf: it's stored as
what it is.
$ curl -i http://127.0.0.1:8000/documents -F files=@invoice.pdf -F 'files=@minutes.docx;filename=minutes.pdf'
HTTP/1.1 201 Created
[{"id":"1f8b5cef-0d26-48e2-b6d0-03cdd086c6f0","name":"invoice.pdf","content_type":"application/pdf","size":142,"width":null,"height":null},{"id":"5a667cf4-565c-4e7a-88ef-ae6a014c1761","name":"minutes.pdf","content_type":"application/vnd.openxmlformats-officedocument.wordprocessingml.document","size":540,"width":null,"height":null}]
And it's served as a Word document, as an attachment, with the extension that says so:
$ curl -D - -o minutes.docx http://127.0.0.1:8000/files/5a667cf4-565c-4e7a-88ef-ae6a014c1761
HTTP/1.1 200 OK
content-type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
content-disposition: attachment; filename="minutes.docx"
content-length: 540
x-content-type-options: nosniff
content-security-policy: default-src 'none'; frame-ancestors 'none'
What to look at¶
- The type comes from the bytes:
CheckedFile.content_typeis what the file is, whatever its name or the client'sContent-Type. It's stored with the file and used to serve it, together withnosniff, so a browser can't be talked into reading an upload as HTML. - Names of the app's own: each file is stored as
{uuid}{CheckedFile.extension}. The client's name (CheckedFile.filename, already made safe) is kept for display, and the download name goes throughsafe_filenameagain with the real extension: no path, no control or invisible characters (such as the right-to-left override),_before the names Windows reserves. - Two size limits, on purpose: the multipart parser stores the whole request before an
endpoint runs, so
BodySizeLimitMiddlewarebounds it per route (a little more than the files allowed, for the multipart framing; 64 KiB anywhere else).FastAPIUploadsthen counts each file exactly, and checks the pixels of images. - All or nothing:
/documentschecks every file before it stores any, so one bad file in a request leaves nothing half-saved. - Attachments for documents: a PDF can carry scripts, so documents are downloaded rather than opened by the browser; images are shown inline.
SecurityHeadersMiddlewarelast: added last, it's the outermost middleware, so the 413s ofBodySizeLimitMiddlewareget the security headers too.
The code¶
examples/uploads/main.py
"""Avatars and documents uploaded, checked by their bytes, stored and served back safely.
uvicorn examples.uploads.main:app
- `POST /avatars`: one image (PNG, JPEG, GIF or WebP) of up to 2 MiB and 16 million pixels,
checked by `FastAPIUploads` as a dependency.
- `POST /documents`: up to five PDF, Word, Excel or PowerPoint files of up to 10 MiB each, all
checked with `check()` before any of them is stored.
- `GET /files/{file_id}`: a file back, with the type found in its bytes, never the one the
client said, and `X-Content-Type-Options: nosniff`. Images inline; documents as attachments,
under the client's name made safe, with the extension of what the file really is.
A file is judged by its bytes, not by its name or `Content-Type`: an HTML page sent as `cat.png`
gets a 415. Files are stored under names of the app's own. `BodySizeLimitMiddleware` gives each
route a limit, so that the multipart parser doesn't store a huge body before the checks run.
"""
import shutil
import tempfile
from collections.abc import AsyncIterator
from contextlib import ExitStack, asynccontextmanager
from pathlib import Path
from typing import Annotated, BinaryIO
from uuid import UUID, uuid4
import uvicorn
from fastapi import Depends, FastAPI, File, HTTPException, Request, UploadFile, status
from fastapi.responses import FileResponse
from pydantic import BaseModel, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
from starlette.concurrency import run_in_threadpool
from fastapi_alcazaba.security import BodySizeLimitMiddleware, SecurityHeadersMiddleware
from fastapi_alcazaba.uploads import IMAGES, CheckedFile, FastAPIUploads, safe_filename
from fastapi_alcazaba.uploads.checks import OFFICE_DOCUMENTS
MIB = 1024 * 1024
MAX_DOCUMENTS = 5
avatars = FastAPIUploads(types=IMAGES, max_size=2 * MIB, max_pixels=16_000_000)
# Without the macro-enabled Office types (.docm, .xlsm, .pptm): OFFICE_DOCUMENTS leaves them out.
documents = FastAPIUploads(types={"application/pdf", *OFFICE_DOCUMENTS}, max_size=10 * MIB)
class StoredFile(BaseModel):
id: UUID
key: str = Field(exclude=True) # where it's stored: its id and the extension of its type
name: str # the client's, made safe: for display only
content_type: str # found in its bytes
size: int
width: int | None = None
height: int | None = None
class Storage:
"""Files in a directory, under names of the app's own. Their details are kept in memory,
where a real app has a database table (and maybe object storage, such as S3, for the
files)."""
def __init__(self, directory: Path) -> None:
self.directory = directory
self.files: dict[UUID, StoredFile] = {}
async def save(self, upload: CheckedFile) -> StoredFile:
file_id = uuid4()
stored = StoredFile(
id=file_id,
key=f"{file_id}{upload.extension}", # never the client's name
name=upload.filename,
content_type=upload.content_type,
size=upload.size,
width=upload.width,
height=upload.height,
)
await run_in_threadpool(self._write, self.directory / stored.key, upload.file.file)
self.files[file_id] = stored
return stored
@staticmethod
def _write(path: Path, source: BinaryIO) -> None:
with path.open("xb") as target: # "x": never overwrite a file
shutil.copyfileobj(source, target)
class Settings(BaseSettings):
"""The configuration, from environment variables or a .env file."""
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
upload_dir: Path | None = None
@asynccontextmanager
async def lifespan(application: FastAPI) -> AsyncIterator[None]:
with ExitStack() as stack:
# UPLOAD_DIR in production: a volume that every process shares. For local development, a
# throwaway directory, deleted at shutdown.
directory = Settings().upload_dir or stack.enter_context(
tempfile.TemporaryDirectory(prefix="uploads-")
)
application.state.storage = Storage(Path(directory))
yield
app = FastAPI(lifespan=lifespan)
# Uploads only where they're expected, with room for the multipart framing around the files.
app.add_middleware(
BodySizeLimitMiddleware,
max_size=64 * 1024,
paths={"/avatars": 2 * MIB + 16 * 1024, "/documents": MAX_DOCUMENTS * 10 * MIB + 64 * 1024},
)
app.add_middleware(SecurityHeadersMiddleware) # last, the outermost: the 413s get the headers too
def get_storage(request: Request) -> Storage:
storage: Storage = request.app.state.storage
return storage
StorageDep = Annotated[Storage, Depends(get_storage)]
@app.post("/avatars", status_code=status.HTTP_201_CREATED)
async def upload_avatar(
upload: Annotated[CheckedFile, Depends(avatars)],
storage: StorageDep,
) -> StoredFile:
"""One image, in the form field `file`: 415 if it isn't one, 413 if it's too large."""
return await storage.save(upload)
@app.post("/documents", status_code=status.HTTP_201_CREATED)
async def upload_documents(
files: Annotated[list[UploadFile], File()],
storage: StorageDep,
) -> list[StoredFile]:
"""Several documents, in the form field `files`: all checked before any is stored, so that
one bad file stores none."""
if len(files) > MAX_DOCUMENTS:
raise HTTPException(422, f"At most {MAX_DOCUMENTS} files at a time")
checked = [await documents.check(file) for file in files]
return [await storage.save(upload) for upload in checked]
@app.get("/files/{file_id}")
async def download(file_id: UUID, storage: StorageDep) -> FileResponse:
"""The file, as the type found in its bytes. A real app checks who may read it."""
stored = storage.files.get(file_id)
if stored is None:
raise HTTPException(404, "File not found")
# The client's name with the extension of the real type: a Word document sent as
# `minutes.pdf` is downloaded as `minutes.docx`, and opens in the right program.
stem = stored.name.rpartition(".")[0] or stored.name
return FileResponse(
storage.directory / stored.key,
media_type=stored.content_type,
filename=safe_filename(stem + Path(stored.key).suffix),
# A PDF can carry scripts: the browser saves documents rather than open them.
content_disposition_type="inline" if stored.content_type in IMAGES else "attachment",
)
if __name__ == "__main__":
uvicorn.run("main:app", reload=True)