Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UrlForm decoding support #1113

Merged
merged 43 commits into from
Aug 16, 2023
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
7ab3c8d
Update tests to cover decoding, fix various bugs uncovered by them
dhpiggott Jul 27, 2023
e20d0d3
Consolidate cursor behaviour within UrlFormCursor
dhpiggott Aug 2, 2023
5d0e251
Remove redundant FailedValue cursor type
dhpiggott Aug 2, 2023
7accb71
Use headOption, not head
dhpiggott Aug 2, 2023
c4d1ae6
Further cursor simplification
dhpiggott Aug 2, 2023
bc174d2
Handle renaming TODOs
dhpiggott Aug 2, 2023
a5fc091
Flatten UrlForm types by making encoder and decoder cursor work with …
dhpiggott Aug 2, 2023
5ceae82
Flatten UrlFormCursor
dhpiggott Aug 2, 2023
e2466f5
Handle TODO re. DRYing error messages
dhpiggott Aug 2, 2023
679c682
Remove TODO
dhpiggott Aug 2, 2023
c100080
Make spec test primitive decoding too
dhpiggott Aug 2, 2023
e99c5cb
Remove redundant UrlFormParserSpec
dhpiggott Aug 2, 2023
06cf841
Fix codecs
dhpiggott Aug 2, 2023
e1c8cd0
First pass of OAuthCodecs
dhpiggott Aug 2, 2023
149f826
First pass of sandbox clients and servers
dhpiggott Aug 3, 2023
b60496d
Fix copyright headers
dhpiggott Aug 8, 2023
83d6d3c
Apply suggestions from code review
dhpiggott Aug 8, 2023
a96480c
Scalafmt
dhpiggott Aug 8, 2023
2701b0f
Remove redundant dependency
dhpiggott Aug 8, 2023
5ac670f
Convert oauth-sandbox apps to Cats Effect
dhpiggott Aug 8, 2023
1f43fd4
Move OAuthCodecs to http4s module
dhpiggott Aug 8, 2023
ddb8f30
Remove redundant TODO
dhpiggott Aug 9, 2023
4ed323e
Remove TODO re. PayloadCodec
dhpiggott Aug 9, 2023
0e97a74
Remove redundant TODO and fix bug in Blob
dhpiggott Aug 9, 2023
9068eae
Revert previous bug fix, add test, verify it fails
dhpiggott Aug 9, 2023
4a42267
Reapply bug fix, verify new test passes
dhpiggott Aug 9, 2023
ecfbeee
Fold UrlFormParser into UrlForm, restrict internals visibility
dhpiggott Aug 9, 2023
7f9e8d0
Scalafmt
dhpiggott Aug 9, 2023
58d39eb
Update TODOs re. downstreaming
dhpiggott Aug 9, 2023
40b6754
Remove queryFlattened and queryName
dhpiggott Aug 10, 2023
ed9b4dc
Make URL form support work using urlFormFlattened and urlFormName
dhpiggott Aug 10, 2023
e4337de
Fix OAuthCodecs
dhpiggott Aug 10, 2023
841fa69
Merge branch 'series/0.18' into url-form-decoding-support
dhpiggott Aug 11, 2023
eccb3b7
Fix enum handling
dhpiggott Aug 11, 2023
824c479
Remove OAuthCodecs and oauth-sandbox
dhpiggott Aug 11, 2023
dfe3eb2
Fix aws-sandbox directory
dhpiggott Aug 11, 2023
ffba6d7
Merge branch 'series/0.18' into url-form-decoding-support
dhpiggott Aug 14, 2023
e8d0a89
Fix imports
dhpiggott Aug 14, 2023
8c081b6
Use locally for readability
dhpiggott Aug 14, 2023
e28d74d
Use groupMap
dhpiggott Aug 14, 2023
18a723d
s/list/vector/
dhpiggott Aug 14, 2023
b0ad4eb
Fix use of groupMap for Scala 2.12
dhpiggott Aug 14, 2023
033173c
Oops, too hasty with "dead" code removal
dhpiggott Aug 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ lazy val allModules = Seq(
decline,
codegenPlugin,
benchmark,
sandbox,
`aws-sandbox`,
protocol,
protocolTests,
`aws-kernel`,
Expand Down Expand Up @@ -910,8 +910,8 @@ lazy val benchmark = projectMatrix
.jvmPlatform(List(Scala213), jvmDimSettings)
.settings(Smithy4sBuildPlugin.doNotPublishArtifact)

lazy val sandbox = projectMatrix
.in(file("modules/sandbox"))
lazy val `aws-sandbox` = projectMatrix
.in(file("modules/aws-sandbox"))
.dependsOn(`aws-http4s`)
.settings(
Compile / allowedNamespaces := Seq(
Expand All @@ -922,14 +922,14 @@ lazy val sandbox = projectMatrix
// Ignore deprecation warnings here - it's all generated code, anyway.
scalacOptions ++= Seq(
"-Wconf:cat=deprecation:silent"
) ++ scala3MigrationOption(scalaVersion.value),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't remember what this was doing now but I think it started causing a problem, and didn't seem to be necessary.

),
smithy4sDependencies ++= Seq(
"com.disneystreaming.smithy" % "aws-cloudwatch-spec" % "2023.02.10",
"com.disneystreaming.smithy" % "aws-ec2-spec" % "2023.02.10"
),
libraryDependencies ++= Seq(
Dependencies.Http4s.emberClient.value,
Dependencies.slf4jNop
Dependencies.Slf4jSimple % Runtime
),
run / fork := true
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ import cats.effect.Concurrent
import cats.syntax.all._
import fs2.compression.Compression
import smithy.api.XmlName
import smithy4s.Endpoint
import smithy4s._
import smithy4s.http._
import smithy4s.http4s.kernel._
import smithy4s.xml.internals.XmlStartingPath

private[aws] object AwsEcsQueryCodecs {

Expand All @@ -41,13 +42,8 @@ private[aws] object AwsEcsQueryCodecs {
// These are set to fulfil the requirements of
// https://smithy.io/2.0/aws/protocols/aws-ec2-query-protocol.html?highlight=ec2%20query%20protocol#query-key-resolution.
// without UrlFormDataEncoderSchemaVisitor having to be more aware
// than necessary of these protocol quirks (having it be aware of
// XmlName and XmlFlattened already feels like too much - perhaps in
// a future change UrlFormDataEncoderSchemaVisitor can work with
// better-named hints, and we can use this same transformation
// approach in AwsQueryCodecs to translate the AWS XML hints to
// those new hints).
ignoreXmlFlattened = true,
// than necessary of these protocol quirks.
ignoreUrlFormFlattened = true,
capitalizeStructAndUnionMemberNames = true,
action = endpoint.id.name,
version = version
Expand All @@ -56,22 +52,16 @@ private[aws] object AwsEcsQueryCodecs {
// This pre-processing works in collaboration with the passing of
// the capitalizeStructAndUnionMemberNames flags to
// UrlFormDataEncoderSchemaVisitor.
smithy4s.schema.Schema.transformHintsTransitivelyK(hints =>
Schema.transformHintsTransitivelyK(hints =>
hints.memberHints.get(Ec2QueryName) match {
case Some(ec2QueryName) =>
hints.addMemberHints(
XmlName(ec2QueryName.value)
)
hints.addMemberHints(XmlName(ec2QueryName.value))

case None =>
hints.memberHints.get(XmlName) match {
case Some(xmlName) =>
hints.addMemberHints(
XmlName(xmlName.value.capitalize)
)

case None =>
hints
hints.addMemberHints(XmlName(xmlName.value.capitalize))
case None => hints
}
}
)
Expand All @@ -91,23 +81,15 @@ private[aws] object AwsEcsQueryCodecs {
AwsXmlCodecs
.responseDecoderCompilers[F]
.contramapSchema(
smithy4s.schema.Schema.transformHintsLocallyK(
_ ++ smithy4s.Hints(
smithy4s.xml.internals.XmlStartingPath(
List(responseTag)
)
)
Schema.transformHintsLocallyK(
_ ++ Hints(XmlStartingPath(List(responseTag)))
)
)
val errorDecoderCompilers = AwsXmlCodecs
.responseDecoderCompilers[F]
.contramapSchema(
smithy4s.schema.Schema.transformHintsLocallyK(
_ ++ smithy4s.Hints(
smithy4s.xml.internals.XmlStartingPath(
List("Response", "Errors", "Error")
)
)
Schema.transformHintsLocallyK(
_ ++ Hints(XmlStartingPath(List("Response", "Errors", "Error")))
)
)

Expand All @@ -122,14 +104,15 @@ private[aws] object AwsEcsQueryCodecs {
val shapeName = alt.schema.shapeId.name
alt.hints.get(AwsQueryError).map(_.code).map(_ -> shapeName)
}.toMap
(errorCode: String) => mapping.getOrElse(errorCode, errorCode)
errorCode => mapping.getOrElse(errorCode, errorCode)
}
val errorDiscriminator = AwsErrorTypeDecoder
.fromResponse(errorDecoderCompilers)
.andThen(_.map(_.map {
case HttpDiscriminator.NameOnly(name) =>
HttpDiscriminator.NameOnly(errorNameMapping(name))
case other => other
case other =>
other
}))

val make = UnaryClientCodecs.Make[F](
Expand Down
92 changes: 54 additions & 38 deletions modules/aws-http4s/src/smithy4s/aws/internals/AwsQueryCodecs.scala
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,21 @@ package smithy4s.aws
package internals

import _root_.aws.protocols.AwsQueryError
import alloy.UrlFormFlattened
import alloy.UrlFormName
import cats.effect.Concurrent
import cats.syntax.all._
import fs2.compression.Compression
import org.http4s.EntityEncoder
import smithy4s.Endpoint
import smithy.api.XmlFlattened
import smithy.api.XmlName
import smithy4s._
import smithy4s.codecs.PayloadPath
import smithy4s.http.Metadata
import smithy4s.http._
import smithy4s.http4s.kernel._
import smithy4s.kinds.PolyFunction
import smithy4s.schema.CachedSchemaCompiler
import smithy4s.Blob
import smithy4s.xml.internals.XmlStartingPath

private[aws] object AwsQueryCodecs {

Expand All @@ -48,7 +51,7 @@ private[aws] object AwsQueryCodecs {
)
val requestEncoderCompilersWithCompression = transformEncoders(
requestEncoderCompilers[F](
ignoreXmlFlattened = false,
ignoreUrlFormFlattened = false,
capitalizeStructAndUnionMemberNames = false,
action = endpoint.id.name,
version = version
Expand All @@ -61,23 +64,15 @@ private[aws] object AwsQueryCodecs {
AwsXmlCodecs
.responseDecoderCompilers[F]
.contramapSchema(
smithy4s.schema.Schema.transformHintsLocallyK(
_ ++ smithy4s.Hints(
smithy4s.xml.internals.XmlStartingPath(
List(responseTag, resultTag)
)
)
Schema.transformHintsLocallyK(
_ ++ Hints(XmlStartingPath(List(responseTag, resultTag)))
)
)
val errorDecoderCompilers = AwsXmlCodecs
.responseDecoderCompilers[F]
.contramapSchema(
smithy4s.schema.Schema.transformHintsLocallyK(
_ ++ smithy4s.Hints(
smithy4s.xml.internals.XmlStartingPath(
List("ErrorResponse", "Error")
)
)
Schema.transformHintsLocallyK(
_ ++ Hints(XmlStartingPath(List("ErrorResponse", "Error")))
)
)
// Takes the `@awsQueryError` trait into consideration to decide how to
Expand All @@ -91,14 +86,15 @@ private[aws] object AwsQueryCodecs {
val shapeName = alt.schema.shapeId.name
alt.hints.get(AwsQueryError).map(_.code).map(_ -> shapeName)
}.toMap
(errorCode: String) => mapping.getOrElse(errorCode, errorCode)
errorCode => mapping.getOrElse(errorCode, errorCode)
}
val errorDiscriminator = AwsErrorTypeDecoder
.fromResponse(errorDecoderCompilers)
.andThen(_.map(_.map {
case HttpDiscriminator.NameOnly(name) =>
HttpDiscriminator.NameOnly(errorNameMapping(name))
case other => other
case other =>
other
}))

val make = UnaryClientCodecs.Make[F](
Expand All @@ -112,43 +108,63 @@ private[aws] object AwsQueryCodecs {
}

def requestEncoderCompilers[F[_]: Concurrent](
ignoreXmlFlattened: Boolean,
ignoreUrlFormFlattened: Boolean,
capitalizeStructAndUnionMemberNames: Boolean,
action: String,
version: String
): CachedSchemaCompiler[RequestEncoder[F, *]] = {
val urlFormEntityEncoderCompilers = UrlForm
.Encoder(
ignoreXmlFlattened = ignoreXmlFlattened,
ignoreUrlFormFlattened = ignoreUrlFormFlattened,
capitalizeStructAndUnionMemberNames =
capitalizeStructAndUnionMemberNames
)
.mapK(
new PolyFunction[UrlForm.Encoder, EntityEncoder[F, *]] {
def apply[A](fa: UrlForm.Encoder[A]): EntityEncoder[F, A] =
urlFormEntityEncoder[F].contramap((a: A) =>
urlFormEntityEncoder[F].contramap(a =>
UrlForm(
formData = UrlForm.FormData.MultipleValues(
values = Vector(
UrlForm.FormData.PathedValue(PayloadPath("Action"), action),
UrlForm.FormData
.PathedValue(PayloadPath("Version"), version)
) ++ fa.encode(a).formData.values
)
List(
UrlForm.FormData(PayloadPath("Action"), Some(action)),
UrlForm.FormData(PayloadPath("Version"), Some(version))
) ++ fa.encode(a).values
)
)
}
)
RequestEncoder.restSchemaCompiler[F](
metadataEncoderCompiler = Metadata.AwsEncoder,
entityEncoderCompiler = urlFormEntityEncoderCompilers,
// We have to set this so that a body is produced even in the case where a
// top-level struct input is empty. If it wasn't then the contramap above
// wouldn't have the required effect because there would be no UrlForm to
// add Action and Version to (literally no UrlForm value - not just an
// empty one).
writeEmptyStructs = true
)
RequestEncoder
.restSchemaCompiler[F](
metadataEncoderCompiler = Metadata.AwsEncoder,
entityEncoderCompiler = urlFormEntityEncoderCompilers,
// We have to set this so that a body is produced even in the case where
// a top-level struct input is empty. If it wasn't then the contramap
// above wouldn't have the required effect because there would be no
// UrlForm to add Action and Version to (literally no UrlForm value -
// not just an empty one).
writeEmptyStructs = true
)
.contramapSchema(
// The AWS protocol works in terms of XmlFlattened and XmlName hints,
// even for the input side, which is most definitely _not_ XML. Partly
// because that seems odd, but mostly so that the URL form support can
// be completely agnostic of AWS protocol details, they work with their
// own more appropriately named hints - which is what necessitates the
// translation here.
Schema.transformHintsTransitivelyK { hints =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 (and thanks for the comment)

def translateFlattened(hints: Hints): Hints =
hints.memberHints.get(XmlFlattened) match {
case Some(_) => hints.addMemberHints(UrlFormFlattened())
case None => hints
}
def translateName(hints: Hints): Hints =
hints.memberHints.get(XmlName) match {
case Some(XmlName(name)) =>
hints.addMemberHints(UrlFormName(name))
case None => hints
}
(translateFlattened _ andThen translateName _)(hints)
}
)
}

private def urlFormEntityEncoder[F[_]]: EntityEncoder[F, UrlForm] =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,14 @@
* limitations under the License.
*/

package smithy4s
package sandbox
package aws

import cats.effect._
import com.amazonaws.cloudwatch
import com.amazonaws.ec2
import org.http4s.client.middleware._
import org.http4s.client.middleware.RequestLogger
import org.http4s.ember.client.EmberClientBuilder
import smithy4s.aws._

Expand Down Expand Up @@ -72,8 +76,7 @@ object Main extends IOApp.Simple {
.map(
RequestLogger.colored(
logHeaders = true,
logBody = true,
logAction = Some(IO.println _)
logBody = true
)
)
awsCredentialsProvider = new AwsCredentialsProvider[IO]
Expand Down
6 changes: 6 additions & 0 deletions modules/bootstrapped/test/src/smithy4s/BlobSpec.scala
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@ class BlobSpec() extends FunSuite {

test("equals depends on underlying data structure") {
assert(Blob("foo") != Blob(ByteBuffer.wrap("foo".getBytes)))
assert(Blob("foo") == Blob("foo"))
assert(
Blob(ByteBuffer.wrap("foo".getBytes)) == Blob(
ByteBuffer.wrap("foo".getBytes)
)
)
}

test("ByteBufferBlob.toArray is idempotent, instantiation-wise") {
Expand Down
Loading