Created key.pem and cert.pem instead of .key and .cert extension is it right

I have refered the link given in reference and while using that command which is

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365

I have created key.pem and cert.pem so will it work fine or renaming can be a better option or should I delete it or recreate is it essential to have extension .key and .cert will pem not work

try this out: https://www.liquidweb.com/kb/configure-vsftpd-ssl/

It is totally fine to have .pem instead of .key and .cert.
Just makes changes in path that you mention in config file accordingly.

ssl_tlsv1_1=YES
ssl_tlsv1_2=YES

These two parameters give error. What to do?

Remove them or comment them it will work fine

530 Non-anonymous sessions must use encryption.

This error comes when using local ftp client?

Yes, this error will come if you try to connect using same command as you used for connecting to unsecured ftp.
Search about how to connect to FTPS using your device (windows/linux).