Issue #20191 has been updated by mrkn (Kenta Murata). We should investigate the real-world application codebases instead of gems on rubygems.org. I guess there can be unexpectedly many scripts in non-UTF8 encoding historically, such as EUC-JP and Windows-31J. If so, your proposal will break those many non-UTF-8 applications. ---------------------------------------- Misc #20191: Deprecate magic encoding comment https://bugs.ruby-lang.org/issues/20191#change-106304 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- I would like to ask that we deprecate the magic encoding comment, and instead require all source files to be encoded in UTF-8. There would be many benefits to the performance of both the parser and compiler. It would also help to simplify both. For example, right now a string literal in a file encoded in US-ASCII can result in 3 different encodings, depending on its internal bytes. If the file is encoded in UTF-8, it can only be a UTF-8 string. The encoding comment itself is not very commonly used in gems. If you take the top 100 most downloaded gem versions from rubygems.org and look at the resolved encoding of all of the files, you get: - UTF-8: 11554 - ASCII-8BIT: 35 - US-ASCII: 10 For all of the most recent versions of gems on rubygems.org, you get: - UTF-8: 2967421 - US-ASCII: 20130 - ASCII-8BIT: 9237 - ISO-8859-1: 87 - Windows-1252: 45 - Shift_JIS: 32 - Windows-31J: 22 - Windows-1251: 15 - EUC-JP: 11 - GBK: 4 - KOI8-R: 3 - ISO-8859-15: 2 - UTF8-MAC: 1 - invalid: 33 Note that "invalid" here could have worked on some rubies < 3.2 if they used Encoding#replicate. If we were to change this, the main breaking change concern would be the encoding of strings and symbols that would leave the context of the file by virtue of a constant read/method call. That's why I think it should first be deprecated in a minor release, then removed in the next major. At the moment this would mean for the top 100 gems we would be worried about 0.39% of files, and on rubygems.org as a whole we would be worried about 0.99% of files. If deprecating the entire encoding comment is unacceptable from a compatibility point of view, I would suggest we try only allowing UTF-8, US-ASCII, and ASCII-8BIT. This would still have a lot of value/simplifications/performance opportunities, at the expense of still needing to be parsed and checked. On the top 100 gems this would mean no files would have to change, and on rubygems.org as a whole it would mean we would be worried about 0.009% of files. That being said, if we're going to deprecate this at all, we should probably just do it all the way to get the full benefit. (In case you want to check the math, the script used to calculate these is attached.) ---Files-------------------------------- gems.rb (4.33 KB) -- https://bugs.ruby-lang.org/