Amazon has been steadily improving their CloudFront CDN offering with WAF (Web Application Firewall) capabilities. This is a great feature, however it's ineffective if origin servers can be attacked directly, bypassing CloudFront. With a little extra work, access to the origin can be restricted.
The solution is to add a secret header value at the edge, and configure the load balancer to block requests that are missing this secret. This is necessary because CloudFront distributions are not associated with security groups, nor are fixed IPs available (unlike higher-priced competitors like Kona Site Shield).
First, navigate to Cloudfront → <DISTRIBUTION> → Behaviors → Default → Edit:
Make sure Redirect HTTP to HTTPS is checked. When combined with the Match viewer policy below, this will make sure the secret is always transmitted over HTTPS.
Repeat this step for for all behaviors.
Now add the header secret to the CloudFront distribution. Navigate to Cloudfront → <DISTRIBUTION> → Origins → Edit, and configure the origin with a custom header. The header value should be a long random string.
Visit Services → WAF and Shield → String Matching → Create condition. Create a string condition that matches the header that was set in the previous step.
Next visit Web ACLs → Create web ACL. Choose the region, and associate the ACL with the load balancer. Add a rule using the string match condition from the previous step. Then, set the default rule to block all. The finished ACL should look like this:
Last, configure the security group to only allow traffic from the load balancer, plus any other trusted networks. ICMP Destination Unreachable should also be allowed to make sure jumbo packets and MTU discovery works correctly.
Try visiting the the address of the app servers, and load balancer directly. The app server should time out, and the load balancer should return a 403, while the site should continue operating normally via CloudFront.
That's it!
A few variations are possible; this can be done without an ALB or HTTPS, but both variations would weaken security. ALBs can take a lot of abuse, and without them, the app server would have to implement the header block rule. For me, it's worth the extra few bucks, even if the load balancer isn't otherwise needed.