Stratus Alert 3315

Voter Error events on ftServer, check Rockwell FTA and related SW

Summary

Voter error events have been reported by ftServer systems running virtual machines that include Rockwell SW products which perform license verification using FactoryTalk Activation Manager.

Servers that exhibit voter events should be checked to see if suspect versions of FTAStub.dll is installed on any VM, before any replacement hardware is considered. Licensing sub-component of FTAStub.dll (v7.07.00.0015, v7.07.00.0016, v7.08.00.0002, v7.08.00.0003) used by other Rockwell SW products generates the voter event when communicating with any version of CodeMeter service.

UPDATE 1 (Nov23,2020) : Included new deployment plan from Rockwell Automation.

UPDATE 2 (Nov30,2020) : Included ThinManager update availability

UPDATE 3 (Dec09,2020) : Included RSLinx Classic update availability

Sites Affected

ESXi or Windows Hyper-V ftServer systems running virtual machines that include Rockwell FactoryTalk Activation Manager or ThinManager or RSLinx or RSLogix 5000 (v20.05.00)

Issue

When a specific third party DLL is identified as the cause of voter event for Stratus system, that DLL may be executing following type of low level instruction:

(a) A DLL may be executing a low level special instruction (example: RXRAND, random number generator), without doing required check with BIOS whether those special instructions can be executed. This will result in each CPU reporting a different output, and hence Stratus HW detects voter event.

(b) A DLL may be querying low level HW info which are specific to each CPU. Again this will result in each CPU reporting divergent data, and result in voter event detection by Stratus HW.

Standard tests will not normally detect these issues, as developers generally test their products with standard HW that do not lock step.

CodeMeter service opens handles with CmContainer in FTAStub.dll(which is included in products like Rockwell ThinManager or RSLinx or RSLogix 5000). Stored Licenses and encryption engines have been pre-built into this DLL module. These pre-built (and encrypted) component of CodeMeter will not change, when updating CodeMeter runtime versions, and hence exposes for voter error events for Stratus platforms, for any version of CodeMeter.

Known affected versions of FTAStub.dll:

v7.07.00.0015, v7.07.00.0016, v7.08.00.0002, v7.08.00.0003

Rockwell plans to fix affected products, and new release that use known good FTAStub.dll are planned to be released by Rockwell. It is suggested that sites affected by voter events update to newer releases or patch rollups, that can be obtained from Rockwell Automation. It should be noted that these good versions use the same version number (for FTAStub.dll) as affected release, to ensure compatibility. However, good versions are of different size and different date.

NOTE THE DIFFERENCE in Length OR WriteTime for known bad and good dll versions:

Examples of known bad FTAStub.dll

FullName : y:\Program Files (x86)\Rockwell Software\RSLinx\FTAStub.dll
LastWriteTime : 4/2/2018 1:28:54 PM
Length : 1629512
Version : 7.07.00.0015

FullName : C:\Program Files (x86)\Rockwell Software\RSLogix 5000\ENU\v20\Bin\FTAStub.dll
LastWriteTime : 4/2/2018 2:28:56 PM
Length : 1629512
Version : 7.07.00.0015

FullName : C:\Program Files (x86)\Rockwell Software\ThinManager\FTAStub.dll
LastWriteTime : 4/2/2018 3:27:56 PM
Length : 1629512
Version : 7.07.00.0016

Example of new FTAStub.dll

FullName : C:\Program Files (x86)\Rockwell Software\RSLinx\FTAStub.dll
LastWriteTime : 12/8/2020 11:42:33 PM
Length : 937288
Version : 7.07.00.0015

FullName : C:\Program Files\Automation Control Products\ThinManager\FTAStub.dll
LastWriteTime : 03/11/2020 10:22:04
Length : 2367432
Version : 7.10.00.0003

Action Required

[1] If any ftServer systems report voter error events, please verify version of Studio 5000 and CodeMeter products running on any of the VMs of that system. Since Buggrabber will not collect product info from every VM, these need to be directly requested from end user.

Section [A] or [B] of the attachment lists "Windows PowerShell script" commands that can be used to collect this info.

Section [A] can used for single VM. Section [B] can be used for multiple VMs on same host.

Refer Alerts 3250 (CodeMeter) and 3272 (Studio 5000) for related details.

[2] If site is using known good CodeMeter and Studio 5000 versions, next we need to verify whether any other Rockwell product uses a known affected FTAStub.dll version (v7.07.00.0015, v7.07.00.0016, v7.08.00.0002, v7.08.00.0003)

Section [C] of the attachment lists PowerShell command that can be used collect FTAStub.dll details on VMs that have one or more the following products (based on the product listing obtained in step 1)

(a) FactoryTalk Activation Manager

(b) ThinManager

(d) RSLogix 5000

Command output will list product directory location, and the specific version of FTAStub.dll

[3] For every case of VMs using an affected FTAstub.dll version, following steps need to be taken to update this DLL file.

(a) Determine the Rockwell product using affected FTAStub.dll file (v7.07.00.0015, v7.07.00.0016, v7.08.00.0002, v7.08.00.0003)

(b) Update the product to newer version available from Rockwell Automation. Refer "Available Fixes" for related links to download from RA.

ThinManager update (Available)
RSLinx Classic Patch Update for version 4.00 thru 4.20 (Available)
All other Rockwell products (ETA - Late Dec 2020)

See SW_products.txt attachment for PowerShell scripts related to above steps (Sections A, B, C)

Avoidance

See “Action Required”.

Available Fixes

Planned Patch and updates (from Rockwell Automation):

ThinManager v11.02.02 -SP2 (Released - Nov 25)
RSLinx Classic Patch Update for version 4.00 thru 4.20 (Released - Dec 08)
All other Rockwell products (ETA - Late Dec 2020)

File Attachments

File 1 None

File 2 SW_products.txt