ISR(Incremental Static Regeneration) feature not generating dynamic pages in s3 bucket #995

techexe-code · 2021-04-15T15:17:06Z

pages are generated in s3 on this folder[static-pages/{**}.html and _next/data/{id}/*.json] build time only

when I open url like page/3 in my browser it's always show me loading state then comes up main content and this page is not being generated in s3

function Post({ post }) {
  const router = useRouter()
  if (router.isFallback) {
    return <div>Loading...</div>
  }

  // Render post...
}

// This function gets called at build time
export async function getStaticPaths() {
  return {
    // Only `/posts/1` and `/posts/2` are generated at build time
    paths: [{ params: { id: '1' } }, { params: { id: '2' } }],
    // Enable statically generating additional pages
    // For example: `/posts/3`
    fallback: true,
  }
}

// This also gets called at build time
export async function getStaticProps({ params }) {
  const res = await fetch(`https://.../posts/${params.id}`)
  const post = await res.json()

  return {
    props: { post },
    revalidate: 5,
  }
}

export default Post

dphang · 2021-04-15T22:32:39Z

We do have e2e test for fallback page here:

serverless-next.js/packages/e2e-tests/next-app-dynamic-routes/pages/optional-catch-all-ssg-with-fallback/[[...catch]].tsx

Line 38 in 2b9dfd8

{ params: { catch: ["b"] } }

/a and /b are generated at build time, rest are first with fallback page but then data request should generate and save the page/data in S3.

You can try here with some random route: https://d2aheyeo29vn1i.cloudfront.net/optional-catch-all-ssg-with-fallback/[random string], first time it will serve fallback page (which is never cached) and then 2nd time (if you hit the same route) it will serve that page

I am wondering if you are maybe caching the fallback page causing this behavior? What is your min cache TTL? If you set some min cache in CloudFront, it will force CloudFront to cache that irrespective of cache control header (for fallback it is public, max-age=0, s-maxage=0, must-revalidate so it should never be cached unless you have a min TTL on CloudFront)

techexe-code · 2021-04-16T04:09:09Z

Cache behaviour of CloudFront

path pattern : Default (*)
Minimum TTL : 0
Maximum TTL : 31536000
Default TTL : 0

path pattern : _next/data/*
Minimum TTL : 0
Maximum TTL : 31536000
Default TTL : 0

path pattern : _next/static/*
Minimum TTL : 0
Maximum TTL : 31536000
Default TTL : 86400

path pattern : static/*
Minimum TTL : 0
Maximum TTL : 31536000
Default TTL : 86400

dphang · 2021-04-16T07:22:47Z

Strange, then I am not quite sure what is going on..is there any other error in the console/network tab or perhaps some S3 write permission issue?

If you are able, please try to build this component and add logs around here:

serverless-next.js/packages/libs/lambda-at-edge/src/default-handler.ts

Lines 639 to 660 in 2b9dfd8

    
             const s3JsonParams = { 
        
               Bucket: bucketName, 
        
               Key: `${s3BasePath}${jsonKey}`, 
        
               Body: JSON.stringify(renderOpts.pageData), 
        
               ContentType: "application/json", 
        
               CacheControl: "public, max-age=0, s-maxage=2678400, must-revalidate" 
        
             }; 
        
             const s3HtmlParams = { 
        
               Bucket: bucketName, 
        
               Key: `${s3BasePath}${htmlKey}`, 
        
               Body: html, 
        
               ContentType: "text/html", 
        
               CacheControl: "public, max-age=0, s-maxage=2678400, must-revalidate" 
        
             }; 
        
             const { PutObjectCommand } = await import( 
        
               "@aws-sdk/client-s3/commands/PutObjectCommand" 
        
             ); 
        
             await Promise.all([ 
        
               s3.send(new PutObjectCommand(s3JsonParams)), 
        
               s3.send(new PutObjectCommand(s3HtmlParams)) 
        
             ]); 
        
           }

, this is where it writes back the page/data to S3. Or if you are able to create a repro repository with similar settings as your project, it would great help too. Thanks!

techexe-code · 2021-04-16T09:24:36Z

not getting any error in console. I don't think there is s3 write permission issue because pages are getting stored during the deployment.

dphang · 2021-04-16T09:38:15Z

Yes, I was referring to if there is a runtime permission issue of Lambda itself (not AWS key used during deployment, that definitely should be fine)

techexe-code · 2021-04-16T14:22:22Z

Is there any way to debug this issue ?
Working fine in localhost

techexe-code · 2021-04-16T16:16:26Z

I have created new project and deployed it to CloudFront but still same issue

build time pages

https://d3vk8t6m4jw2vm.cloudfront.net/post/first
https://d3vk8t6m4jw2vm.cloudfront.net/post/second

above two pages are served from s3

  // This function gets called at build time
  export async function getStaticPaths() {
    return {
      // Only `/post/first` and `/post/second` are generated at build time
      paths: [{ params: { id:'first' } }, { params: {id:'second'} }],
      // Enable statically generating additional pages
      // For example: `/post/third`
      fallback: true,
    }
  }

  // This also gets called at build time
  export async function getStaticProps({ params }) {
    const post = {'id':params.id};
  
    await new Promise(res => setTimeout(res,3000))

    return {
      props: { post },
      revalidate: 5,
    }
  }

fallback pages
https://d3vk8t6m4jw2vm.cloudfront.net/post/third
https://d3vk8t6m4jw2vm.cloudfront.net/post/xxx

and these pages are not getting stored in s3 and loading on every request

dphang · 2021-04-16T17:42:17Z

I did check your link and your fallback page for some reason is being cached: cache-control: public, max-age=0, s-maxage=2678400, must-revalidate, which would be the cause of it always loading fallback page. However, it should not be per:

serverless-next.js/packages/libs/lambda-at-edge/src/default-handler.ts

Line 740 in f64fec4

    
           ? "public, max-age=0, s-maxage=0, must-revalidate" // fallback should never be cached

Are you using the latest alpha version? If so, I believe this might have been fixed. I forgot to check which version you were on, in the future please include that so we can verify if it was already fixed.

If you are still having the same issue on latest version, I am not sure what is going on, so either:

You will need to either debug this component yourself by adding log statements / etc. to the code I pointed (default handler) and build and deploy it.
If you can share a GitHub repo that can reproduce the issue on the latest alpha version, I can take a look at it.

techexe-code · 2021-04-17T03:38:54Z

No, I am using 1.18.0 version

@sls-next/[email protected]

techexe-code · 2021-04-17T04:43:28Z

Thanks again @dphang

I have changed version to @sls-next/[email protected] and Yes, now page is being generated in s3 but has some problem with revalidation.

techexe-code · 2021-04-18T04:21:53Z

@sls-next/[email protected]

I am facing problem with the page revalidation feature of ISR.
I have declared x seconds for revalidation of page but page is still used old content and not getting updated.

This issue is with @sls-next/[email protected] or from my side ?

dphang · 2021-04-18T04:25:31Z

It's not supported yet unfortunately... #804 I am working on releasing 1.19 this month and then hopefully we can take it up. Looks like there is a bounty too

techexe-code · 2021-04-18T04:34:26Z

Great, I really appreciate you and waiting for next release. 😊

adamdottv · 2021-04-19T00:10:53Z

Looks like there is a bounty too

There is! I'm offering a $2000 bounty for ISR support.

dphang · 2021-04-19T00:39:47Z

@adamelmore great! That definitely sounds like a good incentive! I would do myself but unfortunately don't have enough time to take it on yet until I finish existing bugs / releasing 1.19 (I don't really need the financial incentive myself). But if someone is interested they can start on it and join our Slack. I agree it is an important feature in Next.js but I think it definitely requires several days work to make sure it is well tested.

I will probably try to put that in one of the pinned issues for more visibility.

ignaciolarranaga · 2021-05-02T19:25:55Z

Guys I think there is a problem with the permissions to write the generated pages in the bucket (at least in my case seems to be that - I'm currently testing to give manual permissions to the defaultLambda).

I'm using:

export async function getStaticPaths() {
  return {
    paths: [],
    fallback: 'blocking'
  }
}

export async function getStaticProps(context) {
  let data = await getData({...});

  return {
    props: {
      data
    },
    revalidate: 60 * 60 // Revalidate every hour
  }
}

And Cloudfront fails with 503:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>503 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: ...
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

By digging into the cloudfront logs of my edge (in this case Sao Pablo / sa-east-1) I found:

2021-05-02T18:45:13.072Z	c01c1601-25d7-4d78-b10a-f34654bbc53f	ERROR	Invoke Error 	{
    "errorType": "AccessDenied",
    "errorMessage": "Access Denied",
    "Code": "AccessDenied",
    "RequestId": "E9SESHMFAXW7F0A4",
    "HostId": "e5J++3otEsYvt6eA1WoTk9+aPzQgU+BYV0lvyeufar0fu3LHc3JWwWcPyDwcOtGflQ/MV+6jnPg=",
    "name": "AccessDenied",
    "$fault": "client",
    "$metadata": {
        "httpStatusCode": 403,
        "httpHeaders": {
            "x-amz-request-id": "E9SESHMFAXW7F0A4",
            "x-amz-id-2": "e5J++3otEsYvt6eA1WoTk9+aPzQgU+BYV0lvyeufar0fu3LHc3JWwWcPyDwcOtGflQ/MV+6jnPg=",
            "content-type": "application/xml",
            "transfer-encoding": "chunked",
            "date": "Sun, 02 May 2021 18:45:12 GMT",
            "server": "AmazonS3",
            "connection": "close"
        },
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "stack": [
        "AccessDenied: Access Denied",
        "    at deserializeAws_restXmlPutObjectCommandError (/var/task/index-4d1c10fe.js:3789:41)",
        "    at processTicksAndRejections (internal/process/task_queues.js:97:5)"
    ]
}

By the way I'm using the CDK Construct (I don't think it changes but commenting any way).
Version: 1.8.0-alpha.42

kirkness · 2021-05-02T19:38:45Z

Hi @ignaciolarranaga, this will be fixed in #1028, although you can solve right now using something like:

const myApp = new NextJSLambdaEdge()
myApp.bucket.grantReadWrite(myApp.defaultNextLambda);

ignaciolarranaga · 2021-05-02T19:39:17Z

Thank, yes, I was about to comment the same :)

Thanks!

gmontanola · 2021-05-06T14:31:00Z

Hi @ignaciolarranaga, this will be fixed in #1028, although you can solve right now using something like:
const myApp = new NextJSLambdaEdge()
myApp.bucket.grantReadWrite(myApp.defaultNextLambda);

Is there a way of solving this outside the CDK Construct scope?

techexe-code · 2021-05-15T11:06:10Z

@sls-next/[email protected]

I am facing problem with the page revalidation feature of ISR.
I have declared x seconds for revalidation of page but page is still used old content and not getting updated.

This issue is with @sls-next/[email protected] or from my side ?

Is this resolved ?

ignaciolarranaga · 2021-05-15T11:19:54Z

@gmontanola what do you mean by "outside the CDK Construct scope"?.
I'm applying the CDK workaround because I'm creating my stack with CDK, it depends how you create your stack the way that you will solve the issue.

The root issue is permissions, let's say, you have to have to grant permissions to the default lambda to write in the bucket.

dphang · 2021-05-19T04:18:21Z

This can be closed since #1028 was released. Please help try it out and report any bugs in a new issue. Thanks!

techexe-code · 2021-05-21T11:36:39Z

@sls-next/[email protected]

I am facing problem with the page revalidation feature of ISR.
I have declared x seconds for revalidation of page but page is still used old content and not getting updated.

This issue is with @sls-next/[email protected] or from my side ?

Updated Version @sls-next/[email protected]

I need help.
Still, I am facing problem with the page revalidation.

fallback pages :
https://d3vk8t6m4jw2vm.cloudfront.net/post/third
https://d3vk8t6m4jw2vm.cloudfront.net/post/xxx

generated page Cache-Control: public, max-age=0, s-maxage=25, must-revalidate but api is not called after 60 second. May be SQS has some problem.

vitoorgomes · 2021-05-27T14:03:49Z

@sls-next/[email protected]
I am facing problem with the page revalidation feature of ISR.
I have declared x seconds for revalidation of page but page is still used old content and not getting updated.
This issue is with @sls-next/[email protected] or from my side ?

Updated Version @sls-next/[email protected]

I need help.
Still, I am facing problem with the page revalidation.

fallback pages :
https://d3vk8t6m4jw2vm.cloudfront.net/post/third
https://d3vk8t6m4jw2vm.cloudfront.net/post/xxx

generated page Cache-Control: public, max-age=0, s-maxage=25, must-revalidate but api is not called after 60 second. May be SQS has some problem.

I'm facing the same issue, I was testing with some dumb example creating a new Date() on the staticProps and passing to the component and seeing the time revalidate based on the amount of seconds that I set up, this way is working fine, like it was supposed to. But I'm having the same problem as if the function is not calling the API and refreshing the data, it only happens if I do a new deploy. My dev enviroment is outdated because of that, I was looking and the last data that I have is from yesterday when I had to deploy few things but the data is not being updated

I was previously working with the @sls-next/[email protected], already updated and I'm currently using @sls-next/[email protected], but still facing the same problem 😢

techexe-code changed the title ~~incremental-static-regeneration not generating dynamic pages in s3 bucket~~ ISR(Incremental Static Regeneration) feature not generating dynamic pages in s3 bucket Apr 15, 2021

kirkness mentioned this issue Apr 28, 2021

Incremental static regeneration #1028

Merged

13 tasks

dphang closed this as completed May 19, 2021

Uh oh!

ISR(Incremental Static Regeneration) feature not generating dynamic pages in s3 bucket #995

ISR(Incremental Static Regeneration) feature not generating dynamic pages in s3 bucket #995

Comments

techexe-code commented Apr 15, 2021

dphang commented Apr 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

techexe-code commented Apr 16, 2021

Uh oh!

dphang commented Apr 16, 2021

Uh oh!

techexe-code commented Apr 16, 2021

Uh oh!

dphang commented Apr 16, 2021

Uh oh!

techexe-code commented Apr 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

techexe-code commented Apr 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dphang commented Apr 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

techexe-code commented Apr 17, 2021

Uh oh!

techexe-code commented Apr 17, 2021

Uh oh!

techexe-code commented Apr 18, 2021

Uh oh!

dphang commented Apr 18, 2021

Uh oh!

techexe-code commented Apr 18, 2021

Uh oh!

adamdottv commented Apr 19, 2021

Uh oh!

dphang commented Apr 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ignaciolarranaga commented May 2, 2021

Uh oh!

kirkness commented May 2, 2021

Uh oh!

ignaciolarranaga commented May 2, 2021

Uh oh!

gmontanola commented May 6, 2021

Uh oh!

techexe-code commented May 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ignaciolarranaga commented May 15, 2021

Uh oh!

dphang commented May 19, 2021

Uh oh!

techexe-code commented May 21, 2021

Uh oh!

vitoorgomes commented May 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dphang commented Apr 15, 2021 •

edited

Loading

techexe-code commented Apr 16, 2021 •

edited

Loading

techexe-code commented Apr 16, 2021 •

edited

Loading

dphang commented Apr 16, 2021 •

edited

Loading

dphang commented Apr 19, 2021 •

edited

Loading

techexe-code commented May 15, 2021 •

edited

Loading

vitoorgomes commented May 27, 2021 •

edited

Loading