CLOUDFLAREAPI: CF_REDIRECT/CF_TEMP_REDIRECT should dtrt using Single …

…Redirects (#3002)

Co-authored-by: Josh Zhang <jzhang1@stackoverflow.com>

.gitattributes

@@ -29,3 +29,5 @@ go.sum text eol=lf
 *.jpg binary
 *.png
 *.svg
+
+pkg/cloudflare-go/.changelog text eol=crlf

documentation/provider/cloudflareapi.md

@@ -63,6 +63,8 @@ DNSControl requires the token to have the following permissions:
 * Add: Enable SSL controls (`Zone → SSL and Certificates → Edit`)
 * Editing Page Rules?
   * Add: Edit Page Rules (`Zone → Page Rules → Edit`)
+* Creating Redirects?
+  * Add: Edit Dynamic Redirect (`Zone → Dynamic Redirect → Edit`)
 * Managing Cloudflare Workers? (if `manage_workers`: set to `true` or `CF_WORKER_ROUTE()` is in use.)
   * Add: Edit Worker Scripts (`Account → Workers Scripts → Edit`)
   * Add: Edit Worker Scripts (`Zone → Workers Routes → Edit`)
@@ -200,6 +202,84 @@ If a domain does not exist in your Cloudflare account, DNSControl
 will automatically add it when `dnscontrol push` is executed.
 
 
+## Old-style vs new-style redirects
+
+Old-style redirects uses the [Page Rules][https://developers.cloudflare.com/rules/page-rules/] product feature, which is [going away](https://developers.cloudflare.com/rules/reference/page-rules-migration/).  In this mode,
+`CF_REDIRECT` and `CF_TEMP_REDIRECT` functions generate Page Rules.
+
+Enable it using:
+
+```javascript
+var DSP_CLOUDFLARE = NewDnsProvider("cloudflare", {
+    "manage_redirects": true
+});
+```
+
+New redirects uses the [Single Redirects][https://developers.cloudflare.com/rules/url-forwarding/] product feature.  In this mode, 
+`CF_REDIRECT` and `CF_TEMP_REDIRECT` functions generates Single Redirects.
+
+Enable it using:
+
+```javascript
+var DSP_CLOUDFLARE = NewDnsProvider("cloudflare", {
+    "manage_single_redirects": true
+});
+```
+
+{% hint style="warning" %}
+New-style redirects ("Single Redirect Rules") are a new feature of DNSControl
+as of v4.12.0 and may have bugs.  Please test carefully.
+{% endhint %}
+
+
+Conversion mode:
+
+DNSControl can convert from old-style redirects (Page Rules) to new-style
+redirect (Single Redirects). To enable this mode, set both `manage_redirects`
+and `manage_single_redirects` to true.
+
+{% hint style="warning" %}
+The conversion process only handles a few, very simple, patterns.
+See `providers/cloudflare/singleredirect_test.go` for a list of patterns
+supported.  Please file bugs if you find problems. PRs welcome!
+{% endhint %}
+
+In conversion mode, DNSControl takes `CF_REDIRECT`/`CF_TEMP_REDIRECT`
+statements and turns each of them into two records: a Page Rules and an
+equivalent Single Redirects rule.
+
+Cloudflare processes Single Redirects before Page Rules, thus it is safe to
+have both at the same time, and provides an easy way to test the new-style
+rules.  If they do not work properly, use the Cloudflare web-based control
+panel to manually delete the new-style rule to expose the old-style rule. (and
+report the bug to DNSControl!)
+
+You'll find the new-style rule in the Cloudflare control panel.  It will have
+a very long name that includes the `CF_REDIRECT`/`CF_TEMP_REDIRECT` operands
+plus matcher and replacement expressions.
+
+There is no mechanism to easily delete the old-style rules.  Either delete them
+manually using the Cloudflare control panel or wait for Cloudflare to remove
+the old-style Page Rule feature.
+
+Once the conversion is complete, change
+`manage_redirects` to `false` then either delete the old redirects
+via the CloudFlare control panel or wait for Cloudflare to remove support for the old-style feature.
+
+{% hint style="warning" %}
+Cloudflare's announcement says that they will convert old-style redirects (Page Rules) to new-style
+redirect (Single Redirects) but they do not give a date for when this will happen.  DNSControl
+will probably see these new redirects as foreign and delete them.
+
+Therefore it is probably safer to do the conversion ahead of them.
+
+On the other hand, if you let them do the conversion, their conversion may be more correct
+than DNSControl's.  However there's no way for DNSControl to manage them since the automatically-generated name will be different.
+
+If you have suggestions on how to handle this better please file a bug.
+{% endhint %}
+
+
 ## Redirects
 The Cloudflare provider can manage "Forwarding URL" Page Rules (redirects) for your domains. Simply use the `CF_REDIRECT` and `CF_TEMP_REDIRECT` functions to make redirects:
 

go.mod

@@ -4,6 +4,8 @@ go 1.22.1
 
 retract v4.8.0
 
+replace github.com/cloudflare/cloudflare-go => ./pkg/cloudflare-go
+
 require google.golang.org/protobuf v1.34.1 // indirect
 
 require (
@@ -24,7 +26,7 @@ require (
 	github.com/babolivier/go-doh-client v0.0.0-20201028162107-a76cff4cb8b6
 	github.com/billputer/go-namecheap v0.0.0-20210108011502-994a912fb7f9
 	github.com/centralnicgroup-opensource/rtldev-middleware-go-sdk/v4 v4.0.3
-	github.com/cloudflare/cloudflare-go v0.96.0
+	github.com/cloudflare/cloudflare-go v0.97.0
 	github.com/digitalocean/godo v1.116.0
 	github.com/ditashi/jsbeautifier-go v0.0.0-20141206144643-2520a8026a9c
 	github.com/dnsimple/dnsimple-go v1.5.1
@@ -105,7 +107,7 @@ require (
 	github.com/go-logr/logr v1.4.1 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-test/deep v1.0.3 // indirect
-	github.com/goccy/go-json v0.10.2 // indirect
+	github.com/goccy/go-json v0.10.3 // indirect
 	github.com/gofrs/uuid v4.0.0+incompatible // indirect
 	github.com/golang-jwt/jwt/v5 v5.2.1 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
@@ -119,7 +121,7 @@ require (
 	github.com/hashicorp/errwrap v1.1.0 // indirect
 	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
-	github.com/hashicorp/go-retryablehttp v0.7.6 // indirect
+	github.com/hashicorp/go-retryablehttp v0.7.7 // indirect
 	github.com/hashicorp/go-rootcerts v1.0.2 // indirect
 	github.com/hashicorp/go-secure-stdlib/parseutil v0.1.6 // indirect
 	github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect

go.sum

@@ -89,8 +89,6 @@ github.com/centralnicgroup-opensource/rtldev-middleware-go-sdk/v4 v4.0.3/go.mod
 github.com/chzyer/logex v1.2.1/go.mod h1:JLbx6lG2kDbNRFnfkgvh4eRJRPX1QCoOIWomwysCBrQ=
 github.com/chzyer/test v1.0.0/go.mod h1:2JlltgoNkt4TW/z9V/IzDdFaMTM2JPIi26O1pF38GC8=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
-github.com/cloudflare/cloudflare-go v0.96.0 h1:wd+qrnyw+C2eXUUujE6BzFEOREkEfoCvogpO5h33FxI=
-github.com/cloudflare/cloudflare-go v0.96.0/go.mod h1:gLP9fJT8ROgRCjHNKxISNNKeU1JEg2yT5uPEEI8x9Ec=
 github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
 github.com/cpuguy83/go-md2man/v2 v2.0.4 h1:wfIWP927BUkWJb2NmU/kNDYIBTh/ziUX91+lVfRxZq4=
 github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
@@ -154,8 +152,8 @@ github.com/go-test/deep v1.0.3/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3a
 github.com/gobwas/glob v0.2.4-0.20181002190808-e7a84e9525fe h1:zn8tqiUbec4wR94o7Qj3LZCAT6uGobhEgnDRg6isG5U=
 github.com/gobwas/glob v0.2.4-0.20181002190808-e7a84e9525fe/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8=
 github.com/goccy/go-json v0.7.8/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
-github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
-github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
+github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA=
+github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
 github.com/gofrs/uuid v4.0.0+incompatible h1:1SD/1F5pU8p29ybwgQSwpQk+mwdRrXCYuPhW6m+TnJw=
 github.com/gofrs/uuid v4.0.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
 github.com/golang-jwt/jwt v3.2.2+incompatible/go.mod h1:8pz2t5EyA70fFQQSrl6XZXzqecmYZeUEB8OUGHkxJ+I=
@@ -227,8 +225,8 @@ github.com/hashicorp/go-hclog v1.6.3/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVH
 github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk=
 github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
 github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
-github.com/hashicorp/go-retryablehttp v0.7.6 h1:TwRYfx2z2C4cLbXmT8I5PgP/xmuqASDyiVuGYfs9GZM=
-github.com/hashicorp/go-retryablehttp v0.7.6/go.mod h1:pkQpWZeYWskR+D1tR2O5OcBFOxfA7DoAO6xtkuQnHTk=
+github.com/hashicorp/go-retryablehttp v0.7.7 h1:C8hUCYzor8PIfXHa4UrZkU4VvK8o9ISHxT2Q8+VepXU=
+github.com/hashicorp/go-retryablehttp v0.7.7/go.mod h1:pkQpWZeYWskR+D1tR2O5OcBFOxfA7DoAO6xtkuQnHTk=
 github.com/hashicorp/go-rootcerts v1.0.2 h1:jzhAVGtqPKbwpyCPELlgNWhE1znq+qwJtW5Oi2viEzc=
 github.com/hashicorp/go-rootcerts v1.0.2/go.mod h1:pqUvnprVnM5bf7AOirdbb01K4ccR319Vf4pU3K5EGc8=
 github.com/hashicorp/go-secure-stdlib/parseutil v0.1.6 h1:om4Al8Oy7kCm/B86rLCLah4Dt5Aa0Fr5rYBG60OzwHQ=

integrationTest/integration_test.go

@@ -25,6 +25,7 @@ var endIdx = flag.Int("end", -1, "Test index to stop after")
 var verbose = flag.Bool("verbose", false, "Print corrections as you run them")
 var printElapsed = flag.Bool("elapsed", false, "Print elapsed time for each testgroup")
 var enableCFWorkers = flag.Bool("cfworkers", true, "Set false to disable CF worker tests")
+var enableCFRedirectMode = flag.String("cfredirect", "", "cloudflare pagerule tests: default=page_rules, c=convert old to enw, n=new-style, o=none")
 
 func init() {
 	testing.Init()
@@ -65,11 +66,21 @@ func getProvider(t *testing.T) (providers.DNSServiceProvider, string, map[string
 		// use this feature. Maybe because we didn't have the capabilities
 		// feature at the time?
 		if name == "CLOUDFLAREAPI" {
+			items := []string{}
 			if *enableCFWorkers {
-				metadata = []byte(`{ "manage_redirects": true, "manage_workers": true }`)
-			} else {
-				metadata = []byte(`{ "manage_redirects": true }`)
+				items = append(items, `"manage_workers": true`)
 			}
+			switch *enableCFRedirectMode {
+			case "":
+				items = append(items, `"manage_redirects": true`)
+			case "c":
+				items = append(items, `"manage_redirects": true`)
+				items = append(items, `"manage_single_redirects": true`)
+			case "n":
+				items = append(items, `"manage_single_redirects": true`)
+			case "o":
+			}
+			metadata = []byte(`{ ` + strings.Join(items, `, `) + ` }`)
 		}
 
 		provider, err := providers.CreateDNSProvider(name, cfg, metadata)
@@ -1830,6 +1841,14 @@ func makeTests() []*TestGroup {
 
 		// CLOUDFLAREAPI features
 
+		// CLOUDFLAREAPI: Redirects:
+
+		// go test -v -verbose -provider CLOUDFLAREAPI                // PAGE_RULEs
+		// go test -v -verbose -provider CLOUDFLAREAPI -cfredirect=c  // Convert: Convert page rules to Single Redirect
+		// go test -v -verbose -provider CLOUDFLAREAPI -cfredirect=n  // New: Convert old to new Single Redirect
+		// ProTip: Add this to just run this test:
+		//  -start 59 -end 60
+
 		testgroup("CF_REDIRECT",
 			only("CLOUDFLAREAPI"),
 			tc("redir", cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1")),
@@ -1838,32 +1857,32 @@ func makeTests() []*TestGroup {
 
 			// Removed these for speed.  They tested if order matters,
 			// which it doesn't seem to.  Re-add if needed.
-			//clear(),
-			//tc("multipleA",
-			//	cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1"),
-			//	cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
-			//),
-			//clear(),
-			//tc("multipleB",
-			//	cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
-			//	cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1"),
-			//),
-			//tc("change1",
-			//	cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
-			//	cfRedir("cnn.**current-domain-no-trailing**/*", "https://change.cnn.com/$1"),
-			//),
-			//tc("change1",
-			//	cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
-			//	cfRedir("cablenews.**current-domain-no-trailing**/*", "https://change.cnn.com/$1"),
-			//),
+			clear(),
+			tc("multipleA",
+				cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1"),
+				cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
+			),
+			clear(),
+			tc("multipleB",
+				cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
+				cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1"),
+			),
+			tc("change1",
+				cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
+				cfRedir("cnn.**current-domain-no-trailing**/*", "https://change.cnn.com/$1"),
+			),
+			tc("change1",
+				cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
+				cfRedir("cablenews.**current-domain-no-trailing**/*", "https://change.cnn.com/$1"),
+			),
 
-			// TODO(tlim): Fix this test case. It is currently failing.
-			//clear(),
-			//tc("multiple3",
-			//	cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
-			//	cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1"),
-			//	cfRedir("nytimes.**current-domain-no-trailing**/*", "https://www.nytimes.com/$1"),
-			//),
+			// NB(tlim): This test case used to fail but mysteriously started working.
+			clear(),
+			tc("multiple3",
+				cfRedir("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
+				cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1"),
+				cfRedir("nytimes.**current-domain-no-trailing**/*", "https://www.nytimes.com/$1"),
+			),
 
 			// Repeat the above tests using CF_TEMP_REDIR instead
 			clear(),
@@ -1888,14 +1907,23 @@ func makeTests() []*TestGroup {
 				cfRedirTemp("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
 				cfRedirTemp("cablenews.**current-domain-no-trailing**/*", "https://change.cnn.com/$1"),
 			),
-			// TODO(tlim): Fix this test case:
-			//tc("tempmultiple3",
-			//	cfRedirTemp("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
-			//	cfRedirTemp("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1"),
-			//	cfRedirTemp("nytimes.**current-domain-no-trailing**/*", "https://www.nytimes.com/$1"),
-			//),
+			// NB(tlim): This test case used to fail but mysteriously started working.
+			tc("tempmultiple3",
+				cfRedirTemp("msnbc.**current-domain-no-trailing**/*", "https://msnbc.cnn.com/$1"),
+				cfRedirTemp("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1"),
+				cfRedirTemp("nytimes.**current-domain-no-trailing**/*", "https://www.nytimes.com/$1"),
+			),
+		),
+
+		testgroup("CF_REDIRECT_CONVERT",
+			only("CLOUDFLAREAPI"),
+			tc("start301", cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1")),
+			tc("convert302", cfRedirTemp("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1")),
+			tc("convert301", cfRedir("cnn.**current-domain-no-trailing**/*", "https://www.cnn.com/$1")),
 		),
 
+		// CLOUDFLAREAPI: PROXY
+
 		testgroup("CF_PROXY A create",
 			only("CLOUDFLAREAPI"),
 			CfProxyOff(), clear(),

models/record.go

@@ -39,6 +39,7 @@ import (
 //	  CF_REDIRECT
 //	  CF_TEMP_REDIRECT
 //	  CF_WORKER_ROUTE
+//	  CLOUDFLAREAPI_SINGLE_REDIRECT
 //	  CLOUDNS_WR
 //	  FRAME
 //	  IMPORT_TRANSFORM
@@ -51,6 +52,8 @@ import (
 //	  URL301
 //	  WORKER_ROUTE
 //
+// NOTE: All NEW record types should be prefixed with the provider name (Correct: CLOUDFLAREAPI_SINGLE_REDIRECT. Wrong: CF_REDIRECT)
+//
 // Notes about the fields:
 //
 // Name:
@@ -138,6 +141,30 @@ type RecordConfig struct {
 	R53Alias         map[string]string `json:"r53_alias,omitempty"`
 	AzureAlias       map[string]string `json:"azure_alias,omitempty"`
 	UnknownTypeName  string            `json:"unknown_type_name,omitempty"`
+
+	// Cloudflare-specific fields:
+	// When these are used, .target is set to a human-readable version (only to be used for display purposes).
+	CloudflareRedirect *CloudflareSingleRedirectConfig `json:"cloudflareapi_redirect,omitempty"`
+}
+
+// CloudflareSingleRedirectConfig contains info about a Cloudflare Single Redirect.
+//
+//	When these are used, .target is set to a human-readable version (only to be used for display purposes).
+type CloudflareSingleRedirectConfig struct {
+	//
+	Code int `json:"code,omitempty"` // 301 or 302
+	// PR == PageRule
+	PRDisplay     string `json:"pr_display,omitempty"` // How is this displayed to the user
+	PRMatcher     string `json:"pr_matcher,omitempty"`
+	PRReplacement string `json:"pr_replacement,omitempty"`
+	PRPriority    int    `json:"pr_priority,omitempty"` // Really an identifier for the rule.
+	//
+	// SR == SingleRedirect
+	SRDisplay        string `json:"sr_display,omitempty"` // How is this displayed to the user
+	SRMatcher        string `json:"sr_matcher,omitempty"`
+	SRReplacement    string `json:"sr_replacement,omitempty"`
+	SRRRulesetID     string `json:"sr_rulesetid,omitempty"`
+	SRRRulesetRuleID string `json:"sr_rulesetruleid,omitempty"`
 }
 
 // MarshalJSON marshals RecordConfig.

pkg/cloudflare-go/.changelog/1001.txt

@@ -0,0 +1,3 @@
+```release-note:note
+docs: add release notes
+```

pkg/cloudflare-go/.changelog/1002.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+rulesets: fix sni action parameter
+```

pkg/cloudflare-go/.changelog/1003.txt

@@ -0,0 +1,3 @@
+```release-note:dependency
+provider: bumps github.com/urfave/cli/v2 from 2.11.0 to 2.11.1
+```

pkg/cloudflare-go/.changelog/1004.txt

@@ -0,0 +1,7 @@
+```release-note:enhancement
+firewall_rule: automatically paginate `List` results unless `Page` and `PerPage` are provided
+```
+
+```release-note:enhancement
+filter: automatically paginate `List` results unless `Page` and `PerPage` are provided
+```

pkg/cloudflare-go/.changelog/1005.txt

@@ -0,0 +1,3 @@
+```release-note:dependency
+provider: bumps github.com/golangci/golangci-lint from 1.47.1 to 1.47.2
+```

pkg/cloudflare-go/.changelog/1006.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+access_application: fix inability to set bool values to false
+```

pkg/cloudflare-go/.changelog/1008.txt

@@ -0,0 +1,3 @@
+```release-note:dependency
+provider: bumps github.com/golangci/golangci-lint from 1.47.2 to 1.47.3
+```

pkg/cloudflare-go/.changelog/1010.txt

@@ -0,0 +1,3 @@
+```release-note:enhancement
+workers: Add support to upload module workers
+```

pkg/cloudflare-go/.changelog/1014.txt

@@ -0,0 +1,3 @@
+```release-note:enhancement
+workers: Add support for attaching a worker to a domain
+```

pkg/cloudflare-go/.changelog/1016.txt

@@ -0,0 +1,7 @@
+```release-note:enhancement
+firewall_rule: fix double endpoint calls & moving towards common method signature
+```
+
+```release-note:enhancement
+filter: fix double endpoint calls & moving towards common method signature
+```