|
*Note: these are preliminary notes about using Google Cloud Storage as a CDN service*
|
|
*Note: these are notes about using Google Cloud Storage as a CDN service. We use GCS in combination with CloudFlare which means we can get away with some things..*
|
|
|
|
|
|
### Preliminaries
|
|
### Preliminaries
|
|
|
|
|
... | @@ -10,35 +10,39 @@ |
... | @@ -10,35 +10,39 @@ |
|
|
|
|
|
Notes:
|
|
Notes:
|
|
|
|
|
|
* gsutil option: -m multithreaded (higher load+cost, faster)
|
|
* gsutil option: -m multithreaded (higher load+cost but faster)
|
|
* cp options: recursively (-R), compressed (-z), verbose (-v) https://developers.google.com/storage/docs/gsutil/commands/cp
|
|
* cp options: recursively (-R), compressed (-z), verbose (-v) https://developers.google.com/storage/docs/gsutil/commands/cp
|
|
* Note: compressed is necessary to get gzipped delivery. Google Cloud storage does not compress on the fly but should decompress on the fly for old browsers that can't accept decompressed files.
|
|
* Note: compressed is necessary to get gzipped delivery. Google Cloud storage does not compress on the fly but should decompress on the fly for old browsers that can't accept decompressed files.
|
|
|
|
|
|
gsutil -m cp -Rzv mathjax gs://mathjax-cdn/
|
|
gsutil -m cp -Rzv mathjax gs://our-bucket/
|
|
|
|
|
|
This also works within GCS, i.e., bucket to bucket. Note that metadata isn't preserved by default (ACLs etc); use `-p` option to preserve them (*warning*: this incurs higher costs).
|
|
This also works within GCS, i.e., bucket to bucket.
|
|
|
|
|
|
|
|
**Note that metadata isn't preserved by default** (public access, caching, custom headers etc); use `-p` option to preserve them (*warning*: this incurs higher costs).
|
|
|
|
|
|
`gsutil -m cp -Rp gs://bucket1/file1 gs://bucket2/`
|
|
`gsutil -m cp -Rp gs://bucket1/file1 gs://bucket2/`
|
|
|
|
|
|
(You'll still need to set the CORS for each bucket though; see below.)
|
|
(You'll still need to set the CORS for each bucket though; see below.)
|
|
|
|
|
|
### make bucket public
|
|
### make files public
|
|
|
|
|
|
Notes:
|
|
Notes:
|
|
|
|
|
|
* should only be needed once
|
|
|
|
* https://developers.google.com/storage/docs/gsutil/commands/acl#set
|
|
* https://developers.google.com/storage/docs/gsutil/commands/acl#set
|
|
|
|
|
|
gsutil -m acl set -R -a public-read gs://mathjax-cdn
|
|
This has to be done whenever files are uploaded (**TODO**: automate!)
|
|
|
|
|
|
|
|
gsutil -m acl set -R -a public-read gs://our-bucket/path/to/file_or_wildcard
|
|
|
|
|
|
|
|
|
|
### set CORS
|
|
### set CORS
|
|
|
|
|
|
Notes:
|
|
Notes:
|
|
|
|
|
|
* should only be needed once
|
|
* should only be needed once per bucket
|
|
* https://developers.google.com/storage/docs/cross-origin
|
|
* https://developers.google.com/storage/docs/cross-origin
|
|
|
|
|
|
create XML file with
|
|
Create an XML file with
|
|
|
|
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<CorsConfig>
|
|
<CorsConfig>
|
... | @@ -60,8 +64,7 @@ create XML file with |
... | @@ -60,8 +64,7 @@ create XML file with |
|
|
|
|
|
Save as `cors.xml` and run
|
|
Save as `cors.xml` and run
|
|
|
|
|
|
gsutil cors set cors.xml gs://fonts.acmecompany.com/
|
|
gsutil cors set cors.xml gs://our-bucket
|
|
|
|
|
|
|
|
|
|
### mime-type headers
|
|
### mime-type headers
|
|
|
|
|
... | @@ -69,32 +72,23 @@ Notes: |
... | @@ -69,32 +72,23 @@ Notes: |
|
|
|
|
|
* https://developers.google.com/storage/docs/gsutil/commands/setmeta
|
|
* https://developers.google.com/storage/docs/gsutil/commands/setmeta
|
|
* the `**` wildcard includes subdirectories, https://developers.google.com/storage/docs/gsutil/addlhelp/WildcardNames
|
|
* the `**` wildcard includes subdirectories, https://developers.google.com/storage/docs/gsutil/addlhelp/WildcardNames
|
|
* EOT and SVG `content-type` headers are detected correctly during upload.
|
|
* EOT and SVG `content-type` headers are detected correctly during upload to GCS.
|
|
* WOFF is set to the old mime type `x-font/woff`; that works but we decided to go with the future-proof `font-woff`. Either change it as below or [see advice here](https://github.com/GoogleCloudPlatform/gsutil/issues/183#issuecomment-35741417).
|
|
* WOFF is set to the old mime type `x-font/woff`; that works but we decided to go with the future-proof `font-woff`. Either change it as below or [see advice here](https://github.com/GoogleCloudPlatform/gsutil/issues/183#issuecomment-35741417) to do this automatically on upload.
|
|
* For otf:
|
|
* For otf:
|
|
|
|
|
|
gsutil setmeta -r -h "Content-Type:font/opentype" -h "Cache-Control:public, max-age=3600" gs://mathjax-cdn/mathjax/**.otf
|
|
gsutil setmeta -r -h "Content-Type:font/opentype" gs://our-bucket/mathjax/**.otf
|
|
* For otf:
|
|
* For otf:
|
|
|
|
|
|
gsutil setmeta -r -h "Content-Type:application/font-woff" -h "Cache-Control:public, max-age=3600" gs://mathjax-cdn/mathjax/**.woff
|
|
gsutil setmeta -r -h "Content-Type:application/font-woff" gs://our-bucket/mathjax/**.woff
|
|
|
|
|
|
|
|
### Caching
|
|
|
|
|
|
### caching
|
|
**Note** Since we are combining GCS with CloudFlare's caching, we will have short caching times on GCS.
|
|
|
|
|
|
**This needs to be updated to reflect the new setup with GCS+CloudFlare -- shorter caches but additional caching on CloudFlare's end**
|
|
This has to be repeated whenever files get uploaded from local.
|
|
|
|
|
|
This is a one time task. Proposed caching is 3 months for all but `MathJax.js` which gets 1 week.
|
|
$ gsutil -m setmeta -r -h "Cache-Control:public, max-age=3600" gs://our-bucket/mathjax/
|
|
|
|
|
|
$ gsutil -m setmeta -r -h "Cache-Control:public, max-age=7776000" gs://mathjax-cdn/mathjax/
|
|
|
|
$ gsutil -m setmeta -r -h "Cache-Control:public, max-age=3600" gs://mathjax-cdn/mathjax/**/MathJax.js
|
|
|
|
|
|
|
|
This is theoretically risky. Since caching can only be specified after making the file public, it could theoretically happen that MathJax.js is called and cached before the second command runs -- fixing it to a 3 month caching that cannot be undone.
|
|
|
|
|
|
|
|
Well, it didn't happen. So we're good.
|
|
|
|
|
|
|
|
### Logging
|
|
|
|
|
|
|
|
Another one time task. Instructions are at https://developers.google.com/storage/docs/gsutil/commands/logging
|
|
|
|
|
|
|
|
### CNAME
|
|
### CNAME
|
|
|
|
|
... | @@ -102,11 +96,16 @@ To use a CNAME for a bucket, the bucket needs to be named like the CNAME, see ht |
... | @@ -102,11 +96,16 @@ To use a CNAME for a bucket, the bucket needs to be named like the CNAME, see ht |
|
|
|
|
|
You also need to verify domain ownership; see https://support.google.com/webmasters/answer/35179?hl=en.
|
|
You also need to verify domain ownership; see https://support.google.com/webmasters/answer/35179?hl=en.
|
|
|
|
|
|
## TODO
|
|
## Testing
|
|
|
|
|
|
|
|
* CORS allow-origin [vital]
|
|
|
|
* A simple test: switch mathjax.org over to the GCS origin (bucket.storage.googlapis.com) and use saucelabs to check rendering across IE, FF, Chrome, Safari.
|
|
|
|
* Since CORS is set on bucket level, we should not have to worry about it.
|
|
|
|
* font mimetypes [non-vital]
|
|
|
|
* **TODO** write script to `curl -I` all woff and otf files, compare header.
|
|
|
|
|
|
|
|
### Logging
|
|
|
|
|
|
|
|
We don't log on GCS anymore since we combine it with CloudFlare (and get their stats).
|
|
|
|
|
|
* tests.
|
|
For the record, instructions are at https://developers.google.com/storage/docs/gsutil/commands/logging |
|
* CORS [vital]
|
|
|
|
* A simple test: switch mathjax.org over to the origin. user browsershots/browserstack etc to check rendering across IE, FF, Chrome, Safari.
|
|
|
|
* mimetype [non-vital]
|
|
|
|
* write script to `curl -I` all woff and otf files, compare header.
|
|
|
|
* document log analysis |
|
|