HP Security Manager - Troubleshooting Issues

Technical white paper  
HP Security Manager 
Troubleshooting Issues 
 
Table of Contents 
Overview ....................................................................................................................................................... 5 
Log Files and Enabling Debugging ........................................................................................................... 6 
HP Security Manager Log Files Locations (HPSM 3.7 and later) ......................................................... 6 
HP Security Manager Log Files Locations (before HPSM 3.7) ............................................................. 7 
Most frequently used log files ................................................................................................................. 7 
Location and names of configuration files for changing the log level of the different log files ..... 7 
Changing the default log location (including logging into a mapped network drive)........................ 9 
Enabling HP Security Manager debugging (service interactions) .................................................... 10 
Enabling HP Security Manager debugging (Instant-On communication)......................................... 11 
Enabling HP Security Manager debugging (device communication issues) ................................... 12 
Enabling HP Security Manager debugging (Certificate Management) ............................................ 13 
Enabling HP Security Manager debugging Certificate Management: SCEP ................................... 14 
Enabling HP Security Manager debugging Certificate Management: EST ...................................... 15 
Enabling logging for Qualys Policy Compliance .................................................................................. 15 
Enabling HPSM app/system diagnostics logging ............................................................................... 16 
Audit log files ........................................................................................................................................... 16 
Enabling FMEA (Failure Mode and Effect Analysis) logging ............................................................... 16 
Enabling HP Print License Service debugging (HPSM 3.5 and older) ............................................... 17 
Required Network Ports .......................................................................................................................... 19 
Client to Server ports ............................................................................................................................................ 19 
Server to Device ports and to hp.com (for firmware assessments)........................................................ 19 
Devices to Server ports ........................................................................................................................................ 19 
Server to SQL database ports ............................................................................................................................ 20 
Server to Email ports ............................................................................................................................................. 20 
Server to Certificate Authority ports ................................................................................................................ 20 
Licensing ports ........................................................................................................................................................ 20 
HPSM application pool to Server ports ............................................................................................................ 20 

Ports Diagram ........................................................................................................................................... 21 
Changing Firewall Settings and Testing Open Ports ........................................................................ 22 
Proxy for Network Service Blocking Communication....................................................................... 24 
Splunk Dashboard for HPSM ................................................................................................................. 25 
Installation Issues .................................................................................................................................... 25 
Common SQL issues............................................................................................................................... 26 
Database Upgrade Failure ..................................................................................................................... 29 
Using SQL script to repair/upgrade/install the database ................................................................. 30 
Error-Maybe a DB access issue in HPSM_service.log file ................................................................. 32 
Problems Launching Security Manager Web Interface ................................................................... 33 
Web page not displayed while HPSM service is running and connecting to DB ............................. 33 
Bindings/firewall problem .................................................................................................................................... 33 
Browser settings causing issues ....................................................................................................................... 34 
Web page not displayed and HPSM service is automatically stopping ........................................... 35 
ServerConfig could not initialized in HPSM_service.log ....................................................................................... 35 
Invalid length for a Base-64 char array or string in HPSM_service.log.............................................................. 37 
Issues when running HPSM Application Pool ...................................................................................... 38 
As network service - Server Error in “/” Application, Cannot open database........................................ 38 
32-Bit Applications not enabled for HPSM application pool....................................................................... 40 
HTTP 500.19 Internal Server Error with config source reference to X-Frame-Options ...................... 41 
HTTP 500.19 Internal Server Error with error code 0x800700c1............................................................... 42 
As a non-admin user or service account (error 503) .................................................................................... 45 
Access Denied when trying to logon to HPSM .................................................................................... 47 
Licensing Issues ....................................................................................................................................... 48 
Licensing Issues in multi-homed environment: Failed to connect to licensing server ................... 48 
Licensing does not work when HPSM’s hostname is changed after installation ........................... 49 
'Insufficient licenses' error after nightly maintenance task with HPSM 3.6 or newer ................... 52 
'Insufficient licenses' after upgrading to HPSM 3.10 or later ............................................................ 53 
Flexera service Will Not Start (Flexera started and then stopped) .................................................. 55 
‘Failed to retrieve the data’ error in the HPSM dashboard and zero licenses available ................ 56 
Insufficient Licenses in HPSM dashboard and zero licenses available with description of 
starting order of Flexera components ................................................................................................. 57 
Mixing licenses limitations ..................................................................................................................... 58 
Other Licensing Issues before HPSM 3.7 ............................................................................................. 59 
Device Status Errors and Credential Failures.................................................................................... 65 
Credential Management ........................................................................................................................ 68 
SNMP .......................................................................................................................................................................... 68 

Admin (EWS) Password ......................................................................................................................................... 69 
Example of Check Credentials............................................................................................................................ 69 
Device Communication Log Files ....................................................................................................................... 71 
Credentials not Validated or Incorrect ............................................................................................................. 72 
Hanging  or Slow Tasks ........................................................................................................................... 77 
Hung Tasks vs. Slow Tasks .................................................................................................................... 77 
Credential Failed or Network Connection Error Impacting Performance ....................................... 77 
Slow device remediation, EWS Password not verified, ping and SNMP are working fine ............. 79 
Performance Impact of Instant-On Remediations.............................................................................. 79 
Slow performance and reaching max SQL connection pool with Timeouts in HPSM_service.log
 ................................................................................................................................................................... 81 
Upgrade Issues ......................................................................................................................................... 83 
‘Invalid Task: No device specified’ after upgrading to HPSM 3.7 ...................................................... 83 
Web page not displayed, hangs, or Invalid Column name error after upgrading to HPSM 3.7...... 87 
T-SQL Execution command line utility has stopped working error ................................................... 89 
Application Error in Event log while executing the InstallSqlScripts ................................................ 91 
HPSM service stops, http Error 503 in browser, and Invalid column error after upgrading ......... 92 
Task Error:  Internal error in processing when re-running a task after upgrading to HPSM 3.8 
and/or unable to complete a new task. ................................................................................................ 94 
Task Error:  After upgrading to HPSM 3.10 some tasks end up in Task Error .................................. 95 
Growing Database and Nightly Maintenance failing (3.6.1 and older) .............................................. 96 
Growing Database and Nightly Maintenance failing (3.7 and newer) ............................................... 98 
Devices tab shows “Loading…” and is not showing any devices or groups after upgrade ............ 99 
After upgrading to HPSM 3.8 the Start Time of a newly created task is always showing 12:00AM
 ................................................................................................................................................................ 100 
After upgrading to HPSM 3.10 zero devices are displayed in the UI and the event viewer shows 
and error with CertExpiryDate in the description and/or Certificate Expiry Date column missing
 ................................................................................................................................................................ 101 
Certificate Installation Failures ........................................................................................................... 104 
Certificate installation fails on HP DesignJet Z9 ............................................................................. 104 
Certificate installation issues with Microsoft CA ............................................................................ 104 
Certificate installation issues with SCEP.......................................................................................... 107 
Workflow for SCEP .............................................................................................................................................. 107 
ASN1 bad tag in HPSM_service.log file using HPSM 3.7.0 or 3.7.1 and Microsoft NDES (SCEP) .... 108 
‘Bad Request to the Pki Provider’ or Cannot Remediate when using dynamic passwords ............ 109 
Microsoft SCEP fails and InternalServerError in Scep.log ...................................................................... 113 
Sectigo SCEP fails when the intermediate CA certificate is in X.509 format while HPSM 3.8 and 
older is expecting CMS format ........................................................................................................................ 114 
Could not retrieve Challenge password from the SCEP Server in Scep.log and questions marks in 

the logged RESPONSE ....................................................................................................................................... 115 
Could not retrieve Challenge password from the SCEP Server in Scep.log and questions marks in the 
logged RESPONSE .................................................................................................................................................. 115 
Could not retrieve Challenge password from the SCEP Server in Scep.log and when the challenge 
password is visible in the SCEP.log ................................................................................................................ 116 
Incorrect Certificate Authority Server or CA server down/unreachable ..................................... 116 
Incorrect Certificate Authority Name in policy ................................................................................ 118 
CA server is configured to require manual approval of the request ............................................ 118 
The Template Name in the policy for the certificate template is incorrect .................................. 120 
Invalid ID certificate when CRL is unreachable ................................................................................ 121 
CertificateTask: Task Error (HPSM 3.8) when using RSA with SHA1 Signature ........................... 122 
CertificateTask: Task Error (HPSM 3.8) when using ECDSA with Key Length P-256 ................... 123 
HPSM fails to install ID certificate (Cannot Remediate) and access denied in logs, while ID 
certificate did get issued by CA server ............................................................................................. 125 
CN value lists the IP address of the printer instead of the DNS name ......................................... 127 
Certificate Installation Failures with Zebra devices........................................................................ 128 
Cannot Remediate (Required file is not found in the device) ......................................................... 131 
Resolving Zebra device certificate issues ................................................................................................... 132 
Miscellaneous ......................................................................................................................................... 132 
HPSM is recommending a firmware downgrade after a firmware assessment of the HP Color 
LaserJet E55040 .................................................................................................................................. 132 
Exported Reports Are Showing Garbage/Unreadable Text ........................................................... 134 
Unable to select a date when creating a task (null visible) ............................................................. 134 
Cannot Remediate Web Encryption or Active Ciphers (Invalid input; data value not supported)
 ................................................................................................................................................................ 135 
HPSM always displaying Medium Risk for File Erase Mode on some legacy HP devices, such as 
HP LaserJet P3010 ............................................................................................................................... 136 
HPSM is unable to communicate over SNMPv3 with some legacy HP Laserjet devices ........... 138 
Missing hostname for a device even when nslookup provides a response ................................. 140 
Column “Certificate Expiry Date” is missing in HPSM UI ................................................................. 141 
HPSM_service.log file errors ................................................................................................................ 141 
CryptographicException error or ‘Padding is invalid and cannot be removed’ error.................. 141 
‘Unable to read table/ when table is empty. Returning empty lists’ error ..................................... 142 
Using Network Traces for Troubleshooting ...................................................................................... 144 
Using Event Viewer for Troubleshooting ........................................................................................... 144 
Appendix A Following a task in HPSM_service.log........................................................................... 145 
Appendix B Syntax for installDBrmt.bat (HPSM 3.4 and older) ..................................................... 148 
Appendix C Other HP Security Manager Whitepapers and Manuals.......................................... 149 
Supported printers and devices ......................................................................................................... 150 

Appendix D FMEA Error Codes ............................................................................................................ 150

Appendix E Splunk Dashboard for Analyzing HPSM Logs ............................................................. 152

Overview

HP Security Manager (HPSM) is a security compliance software used to assess and remediate a

fleet of devices against a desired security policy. This document provides troubleshooting

techniques to use if issues arise when either installing or running HP Security Manager. As with any

software, challenges can be presented by servers, operating systems, networks, devices, etc.

Security Manager relies upon Microsoft IIS and SQL Server, both of which can present challenges

as well. While this document discusses known issues, new issues are always going to be

encountered. Successful troubleshooting always involves removing variables to determine root

cause. Internet search also an excellent troubleshooting tool as most errors are generic Microsoft

errors and are likely encountered by users of other software.

Log Files and Enabling Debugging

Log files are created during installation or upgrade and when running and using HPSM. By default log

level is set to “INFO” or “ERROR”. Enabling debug logging is sometimes critical for finding potential

issue causes such as hanging or slow tasks.

This next section will show which log locations are available. This will be followed with a section which

explains how to change the log levels for the different log files.

and when to change it

HP Security Manager Log Files Locations (HPSM 3.7 and later)

By default there are three directories for a running HPSM installation with the following directories

and log files:

C:\Program Files (x86)\HP Security Manager\log

EapBinding.log

EapDatPerformance.log

EapDeviceLib.log (device communication/configuration issues)

EapMpsBilling.log

EapNetworkLib.log (device communication/configuration issues)

EapResults.log

EapWppClient.log

EapWsEventing.log

Est.log (empty log, use Est.log in HP Security Manager\PkiProviders\log)

Flexera.log (licensing logging)

HPCM.log (contains only information about starting up HPCM component)

HpsmDatTransactions.log

HPSM_Service.log

HPSM_Service_FMEA.log (in beta, logs tasks only since HPSM 3.10)

HPSM_ServiceApp.log

InstantOn.log

MaintenanceTask.log

McoTranslation.log

Qualys.log

Scep.log (empty log, use Scep.log in HP Security Manager\PkiProviders\log)

C:\Program Files (x86)\HP Security Manager\WebApp\log

HPCM.log (contains only information about UI interaction with HPCM component)

HPSMWeb.log

HPSM_WebAudit.log

Scep.log (empty log, use Scep.log in HP Security Manager\PkiProviders\log)

C:\Program Files (x86)\HP Security Manager\PkiProviders\log

Est.log

HPCM.log (contains information about assess and remediation tasks for HPCM)

Scep.log

The installation or upgrade process is logged in different log files in the following temp directory:

C:\Users\”username”\AppData\Local\Temp

MSIxxxxx.LOG (This file will begin with “MSI” and end with “.LOG” but will contain random characters in

the middle as this is produced by the Microsoft Windows installer)

HPSM_Install_log.txt

HPSM_CheckIIS_log.txt (IIS configuration details)

HP_SM_Install_20230531051417.log (YearMonthDayTime)

HP_SM_UnInstall_20230531051632.log (YearMonthDayTime)

When using remote access, the actual files can be in a subdirectory like Temp/1 or Temp/2.

HP Security Manager Log Files Locations (before HPSM 3.7)

Before HPSM 3.7 the word JetAdvantage was part of the installation directory, which resulted in the

following directories:

C:\Program Files (x86)\HP JetAdvantage Security Manager\log

C:\Program Files (x86)\HP JetAdvantage Security Manager\WebApp\log

C:\Program Files (x86)\HP JetAdvantage Security

Manager\PkiProviders\log

HPSM 3.5 and older was using a third service: the HP Print License Service. This service was

integrated into the HP Security Manager service in HPSM 3.6. A separate log file for the license server

was available in the following location (only used by HPSM 3.5 and older):

C:\ProgramData\HP\HP Print License Service Files\HPPLS.log

Log file(s) created by FlexeraLicensingService (HPSM 3.5 and older)

C:\ProgramData\HP\HP Print License Service Files\Flexera.log

Most frequently used log files

For troubleshooting device specific remediation issues the EapDeviceLib.log, EapNetworkLib.log and

HPSM_Service.log are needed.

For troubleshooting problems with the nightly maintenance, the MaintenanceTask.log is needed.

For troubleshooting problems with instant on, the InstantOn.log is needed.

For troubleshooting problems with tasks, the HPSM_service.log and eventually the EapDevice.lib.log

and EapNetworkLib.log.

For troubleshooting assessment/remediation issues with device certificates the HPCM.logs are

needed and eventually the EapDevice.lib.log and EapNetworkLib.log.

For troubleshooting assessment/remediation issues with device certificates using Scep the Scep.log

and HPCM.logs are needed and eventually the EapDevice.lib.log and EapNetworkLib.log.

For troubleshooting assessment/remediation issues with device certificates using Est the Est.log and

the HPCM.logs are needed and eventually the EapDevice.lib.log and EapNetworkLib.log.

Location and names of configuration files for changing the log

level of the different log files

HPSM has different configuration files for changing the log level of the different log files. This section

explains which config file has to be edited for the different log files. In the next section a more

detailed description of the different log files will be provided.

To enable debugging for non-device specific issues (like problems starting a task, problems starting

the service)

C:\Program Files (x86)\HP Security Manager\HPSM_Service.exe.config

To enable debugging for issues related to device communication.

C:\Program Files (x86)\HP Security Manager\EapLogConfig.xml

Security Manager logs events for who did what and when in two different Audit logs. Log level for this

can be changed in the following file:

C:\Program Files (x86)\HP Security Manager\WebApp\Web.config

For Certificate remediation with EST, Scep the config level can be changed in the following file:

C:\Program Files (x86)\HP Security Manager\HPSM_Service.exe.config

Warning: Making changes in the config level in the following files, will

not

have any impact on the

Scep.log and EST.log file in the HP Security Manager\Webapp\log directory as those log files will

always remain empty

C:\Program Files (x86)\HP Security Manager\

WebApp\PkiProviders\HP.HPCM.EstClient.dll.config

C:\Program Files (x86)\HP Security Manager\

WebApp\PkiProviders\HP.HPCM.ScepClient.dll.config

In the following screenshots of the different directories, you can see in which file the log level for the

different log files can be changed.

Changing the default log location (including logging into a

mapped network drive)

The appender sections in the config files define the details of the log behavior (number of log file,

relative , file location, max file size etc. For example for the HPSM_service.log you can find the

following configuration for local logging:

<conversionPattern value="%date %-5level %logger %identity %exception

[%thread] - %message%newline" />

</layout>

</appender>

It’s possible to use a mapped network drive for logging, if the account running the HPSM service has

full rights on the specified mapped network location. The file value will have to be changed and

LockingModel type, and pseudofile with value true will have to be added to the appender

configuration. See the following example:

<conversionPattern value="%date %-5level %logger %identity %exception

[%thread] - %message%newline" />

</layout>

</appender>

Enabling HP Security Manager debugging (service interactions)

To enable debugging for non-device specific issues (like problems starting a task, problems starting

the service), edit the following lines in the HPSM_service.exe.config file. Make changes highlighted

below in yellow and restart the HP Security Manager service:

<appender-ref ref="ServerAppender" />

</logger>

<appender-ref ref="ServerAppender" />

</logger>

<appender-ref ref="ServerAppender" />

</logger>

<appender-ref ref="ServerAppender" />

</logger>

<appender-ref ref="ServerAppender" />

</logger>

<appender-ref ref="ServerAppender" />

</logger>

<appender-ref ref="ServerAppender" />

</logger>

<appender-ref ref="ServerAppender" />

</logger>

The Data logger (highlighted in green) refers to the interactions between the HPSM service and the

SQL database. INFO level for the Data logger is fine for troubleshooting non-database related issues.

HPSM 3.4 introduced auto group policies. To see the order in which tasks are executed in the debug

logs, the enableTaskSequencingLogging must be set to true.

NOTE: These additional log settings will only be available if the logger value for the logger name

Service is set to DEBUG.

By default, the logs will roll over when they reach 100 MB in size, and each log will only roll over one

time before they will start to be deleted in each subsequent rollover. These values can also be

configured in each of the configuration files mentioned previously by changing the following entries:

Enabling HP Security Manager debugging (Instant-On communication)

From HPSM 3.6 onwards the Instant-On announcements and communication will also be logged in a

separate log file. The same information will remain in the HPSM_service.log file.

To enable debugging for instant-on specific issues (like problems high instant-on load, slow Instant-

On processing, devices not displaying in HPSM), edit the following lines in the

HPSM_service.exe.config file. Make changes highlighted below in yellow and restart the HP Security

Manager service:

<appender-ref ref="Instant-OnAppender" />

</logger>

Enabling HP Security Manager debugging (device communication

issues)

To enable debugging for issues related to device communication, edit the following lines in the

EapLogConfig.xml file. Make changes highlighted below in yellow and restart the HP Security

Manager service.

<!-- **LOGGER DEFINITIONS FOLLOW**

Logging levels represented in order of increasing priority (e.g., ALL

includes everything below it, but excludes INFO):

To turn all logging off use:

<appender-ref ref="EapDeviceLibAppender" />

</logger>

<appender-ref ref="EapDeviceLibAppender" />

<appender-ref ref="EapBindingAppender" />

</logger>

<appender-ref ref="EapDeviceLibAppender" />

</logger>

<appender-ref ref="EapResultsAppender" />

</logger>

<!-- ERRORs in this log file are expected, therefore it is off by

default -->

<appender-ref ref="EapNetworkLibAppender" />

</logger>

<appender-ref ref="EapMpsBillingAppender" />

</logger>

<appender-ref ref="EapWsEventingAppender" />

</logger>

<appender-ref ref="McoTranslationAppender" />

</logger>

Brief explanation of the different log files

HPSM includes the following log file types.

• EapDeviceLibrary - This log shows general information like DAT version, the values returned from

device, which methods will be used, etc.

• Pipeline - This log shows the methods (pipelines) to perform different operations. This helps to

identify errors/exceptions in pipeline steps.

• Results - This gives the names of the config items which are processed.

• EapNetworkLib - This log shows the device communications. This helps to identify the retries,

timeouts, etc.

• McoTranslation - This log file shows methods (pipelines) for custom processes. This helps to

identify errors/exceptions in custom processes. Normally not needed. Only upon request from

support team.

• Bindings -This log shows which interface of the device has been selected. Normally not needed to

enable debug mode for this, only in cases with errors like McoNotFound, McoNotSupported is

listed in other log files.

By default, the logs will roll over when they reach 100 MB in size, and each log will only roll over one

time before they start to become deleted in each subsequent rollover. These values can also be

configured in each of the configuration files mentioned previously by changing the following entries:

In some situations (upon request from HP support), it might be required to know which information is

provided by HPSM to the DAT component within HPSM. This requires changing the log level to

DEBUG for the HpsmDatTransaction. Edit the following lines in the HPSM_service.exe.config file.

Make changes highlighted below in yellow and restart the HP Security Manager service:

<appender-ref ref="HpsmDatTransactionAppender" />

</logger>

The corresponding log file HpsmDatTransactions.log will be updated with the internal device

transaction details.

Enabling HP Security Manager debugging (Certificate Management)

When HPSM is interacting with a Microsoft (Standalone or Enterprise CA), OpenTrust CA or Symantec

CA, it is using the Certificate Management log files (HPCM.log). These log files are in three locations:

C:\Program Files (x86)\HP Security Manager\log

C:\Program Files (x86)\HP Security Manager\PkiProviders\log

C:\Program Files (x86)\HP Security Manager\WebApp\log

Debug logging for ID Certificate requests can be enabled in two files:

C:\Program Files (x86)\HP Security Manager\HPSM_Service.exe.config

C:\Program Files (x86)\HP Security Manager\WebApp\Web.config

Note; when using SCEP, EST or OpenTrust separate additional log files are availabled and will have to

be configured in debug mode.

Make the following changes highlighted:

<appender-ref ref="HPCMAppender"/>

</logger>

The changes in the HPSM_Service.config file require a restart of the service HP Security Manager and

will enable the debug logging for the HPCM.log file in the following directories:

C:\Program Files (x86)\HP Security Manager\log

C:\Program Files (x86)\HP Security Manager\PkiProviders\log

The last file will contain information about the interaction with the CA server.

The changes in the Web.config file require a restart of IIS (or recycle the application pool HPSM) and

will enable the debug logging for the HPCM.log file in the directory:

C:\Program Files (x86)\HP Security Manager\WebApp\log

Enabling HP Security Manager debugging Certificate Management:

SCEP

SCEP debugging can be enabled in the HPSM_Service.exe.config file. Make changes highlighted

below in yellow.

<appender-ref ref="ScepAppender" />

</logger>

The changes in the HPSM_Service.config file require a restart of the service HP Security Manager

and will enable the debug logging for the SCEP.log file in the following directories:

C:\Program Files (x86)\HP Security Manager\log

C:\Program Files (x86)\HP Security Manager\PkiProviders\log

The last file will contain information about the interaction with the SCEP server.

Enabling HP Security Manager debugging Certificate Management:

EST

EST debugging can be enabled in the HPSM_Service.exe.config file. Make changes highlighted

below in yellow.

<appender-ref ref="Appender" />

</logger>

The changes in the HPSM_Service.config file require a restart of the service HP Security Manager

and will enable the debug logging for the Est.log file in the following directories:

C:\Program Files (x86)\HP Security Manager\log

C:\Program Files (x86)\HP Security Manager\PkiProviders\log

NOTE: The EST plug-in is available from HPSM 3.5 onwards.

Enabling logging for Qualys Policy Compliance

HPSM 3.5 offers service integration with Qualys. In Qualys debugging can be enabled in the

HPSM_Service.exe.config file. When enabled log entries will be created when printer assessment

results are sent to Qualys. Passwords are not transmitted to Qualys, only the password

assessment result itself. To enable debugging for the Qualys policy integration, make changes

highlighted below in yellow.

<appender-ref ref="QualysAppender" />

</logger>

The changes in the HPSM_Service.config file require a restart of the service HP Security Manager

and will enable the debug logging for the Qualys.log file in the following directories:

C:\Program Files (x86)\HP Security Manager\log

Enabling HPSM app/system diagnostics logging

It’s also possible to add logging on application level. This is normally not needed and therefore

commented out by default. To enable app logging you have to make the following changes in the

HPSM_service.exe.config file:

1. Add the following line under <appSettings>

2. Uncomment the system.diagnostics by removing the <!- - before <system diagnostics> and - ->

after </system diagnostics>. See screenshot of this section (default value on top) and

uncommented version on the bottom.

Audit log files

Besides the debugging log files, Security Manager logs events for who did what and when in two

different Audit logs:

C:\Program Files (x86)\HP Security Manager\

WebApp\log\HPSM_WebAudit.log

This file will have events that are triggered from the user.

C:\Program Files (x86)\HP Security Manager\log\

HPSM_ServiceApp.log

This file contains events about stopping/starting service and license limitations.

This file will have events that are triggered from HPSM service. An explanation of entries in those log

files, can be found in the whitepaper Reporting, Email Alert Subscriptions & Remediation Summary,

Auditing & Syslog Functionality under the section HPSM User Activity Logging (Auditing)

Enabling FMEA (Failure Mode and Effect Analysis) logging

In HPSM 3.10 a new log file was added: HPSM_service_FMEA.log. It’s disabled by default and if

enabled logging tasks and hpsm service information in FMEA mode. The intention of this log file is to

provide structured log files. Structured logs will be in the form of key-value pair so that the logs can

be easily traced via log management solutions like Splunk. This means that logs will be represented

in a completely different way.

Example:

In order to quickly locate the errors and provide recovery mechanism it’s important to have FMEA

matrix and appropriate error codes logged.

The different error codes will be developed overtime and provided.

In the previous example the code is SM3003, and SM3xxxx means instant-on discovery logging.

The list of FMEA error codes and their corresponding descriptions is available in Appendix D.

By default this logging is fully disabled. To enable it, change the value for enableStructLogs from

false to true in the HPSM_service.exe.config and in the web.config file.

<!--enableStructLogs: If set, additional structured and FMEA logs are

generated-->

If you also want to have debug FMEA log entries, you have to change the FMEA appender to DEBUG

in the HPSM_service.exe.config file.

If you also want to get (sub)task details you have to enable enableSctructLogs as well. log entries,

you have to change the FMEA appender to DEBUG in the HPSM_service.exe.config file.

<appender-ref ref="FmeaAppender" />

</logger>

After making above changes, stop the HPSM service. Restart the hpsm application pool and start the

HPSM service.

Enabling HP Print License Service debugging (HPSM 3.5 and older)

To enable debugging for the HP Print License Service, edit the following lines in the

HP.Print.License.Host.WindowsService.exe.config file in the following directory:

C:\Program Files (x86)\HP JetAdvantage Security Manager\HP Print License

Service

Make changes highlighted below in yellow, then stop the service HP JetAdvantage Security Manager

and then stop the HP Print License Service. Restart first the HP Print License Service and after that

restart the service HP JetAdvantage Security Manager.

<appender name="FileAppender"

type="log4net.Appender.RollingFileAppender">

<param name="File" value="C:\ProgramData\HP\HP Print License

Service\HPPLS.log" />

<param name="ConversionPattern" value="%d [%t] %-2p %c - %m%n"

</layout>

</appender>

<root>

<!--Possible value for level 1.ALL

2.DEBUG

3.INFO

4.WARN

5.ERROR

6.FATAL

7.OFF

-->

<appender-ref ref="FileAppender" />

</root>

</log4net>

Required Network Ports

If a firewall is installed on the computer on which the Security Manager service runs, and the service

will be accessed from the user interface on a remote computer, the firewall must be set to allow

access to the service.

The older Security Manager service listens on port 8002, which must be opened in the firewall to

allow remote access to the service. The new browser-based interface listens on port 7637 be

default. If you do not want to allow remote access to the Security Manager web service for either

version, then you can block the respective ports with a firewall.

The following tables list the ports used by HP Security Manager.

Client to Server ports

Port

Protocol

Service

Notes

7637

(version 3.0+)

TCP

HTTPS

Port set during installation to be used to secure data

between client and HPSM server via browser. This

port may be changed to something else by editing

bindings for the HPSM web site under IIS Manager.

HPSM versions 3.0 and beyond.

8002

(version 2.1.5-)

TCP

WCF-NET.TCP

WCF with message encryption - port used from a

remote client interface to the Security Center

service. HPSM versions 2.1.5 and prior.

Server to Device ports and to hp.com (for firmware assessments)

Port

Protocol

Service

Notes

80 and 8080

TCP

HTTP

Port used for HTTP communication to devices only

when SSL is not supported on the device. Also used to

gather the latest firmware versions from the web if

firmware assessments are enabled and configured to

dynamically retrieve from web.

443

TCP

HTTPS

Port used for secure HTTP communication to devices,

HTTP Web over SSL.

N/A

ICMP

PING

Internet Control Message Protocol - port used to

check if node is active.

161

UDP

SNMP

Simple Network Management Protocol - port used for

many configuration items on devices as well as

discovery of devices.

7627

TCP

SOAP-HTTP

Web service port used to manage communications on

HP FutureSmart devices.

Devices to Server ports

Port

Protocol

Service

Notes

3329

TCP

HP Instant-On

Security

Secure port (uses SSL) used from the device to the

Security Manager service for Instant-On discovered

devices.

Server to SQL database ports

Port

Protocol

Service

Notes

1433

TCP

MS SQL

Standard DB Connection - port used from the Security

Manager service to a remote SQL database with a

default instance. Can be customized in a configuration

file.

1434

UDP

MS SQL Browser

service

Standard connection to SQL browser service to

retrieve the TCP port for the named SQL instance

dynamic

TCP

MS SQL

Standard DB connection to a named SQL instance

using dynamic ports

Server to Email ports

Port

Protocol

Service

Notes

SMTP

Simple Mail Transfer

Protocol

Typical port used for communication to mail server if

Automated Output feature is enabled. Port can be

customized under File, Settings, Automated Output.

Server to Certificate Authority ports

Port

Protocol

Service

Notes

135

TCP

DCOM/RPC

Certificate management - port used between

Security Manager service and CA server.

Random allocated

high TCP ports

above 1024

TCP

DCOM/RPC

Certificate management - port used between

Security Manager service and CA server.

Licensing ports

Port

Protocol

Service

Notes

7000

TCP

HP Print License

Service

Licensing heartbeat – Heartbeat port used between

the Security Manager service and the HP Print

License service. This is the communication between

two services on the same machine and needs to be

open on the incoming and outgoing port (HPSM 3.5

and older)

8888

TCP

HP Print License

Service

Licensing - port used between the Security Manager

service and the HP Print License service. This is the

communication between two services on the same

machine and needs to be open on the incoming and

outgoing port (HPSM 3.5 and older)

27000

TCP

Flexera service

Licensing - port used by the Flexera service

(lmgrd.exe). This port is used to communicate

between the FlexeraLicensingService and the HP

Print License Service. These two services reside on

the same machine and therefore this port needs to

be open for incoming and outgoing communication.

HPSM application pool to Server ports

Port

Protocol

Service

Notes

8003

TCP

HPSM Windows

Service Port

Communication between HPSM application pool and

HPSM service.

Ports Diagram

Changing Firewall Settings and Testing Open Ports

When configuring firewalls, an administrator can either open ports used by the application

(see the tables in

Required Network Ports

) or allow certain program executables access

through the firewall. For the latter, Security Manager includes two separate services

represented by three executables:

C:\Program Files (x86)\HP Security Manager\HPSM_Service.exe

C:\Program Files (x86)\HP Security Manager\HPQ.exe

C:\Program Files (x86)\HP Security Manager\lmgrd.exe

Security Manager primarily uses the following ports to communicate to devices:

• 80 – used for non-encrypted HTTP traffic to device

• 161 – SNMP traffic to device

• 443 – encrypted traffic to device

• 7627 – web services traffic to device

HPSM 3.5 and older also uses the following executable:

C:\Program Files (x86)\HP JetAdvantage Security Manager\HP Print

License Service\HP.Print.License.Host.WindowsService.exe

Therefore, these ports need to be open for Security Manager to effectively manage

devices. Security Manager and Web Jetadmin are very similar in how they communicate to

devices. If Web Jetadmin is managing devices fine without credential failures and you are

certain Security Manager has matching credentials in its Credential Store, as a test you

might install Security Manager on the same machine as Web Jetadmin to determine if port

issues exist on the Security Manager server.

PortQry, a free Microsoft utility, can also be used to determine if ports are open between

the Security Manager server and the device. The Port Query user interface tool makes it

easy to quickly enter an IP Address for a printer and the port to be queried.

Results will be displayed whether the port is Listening or Not Listening.

Proxy for Network Service Blocking Communication

Security Manager runs under the Network Service account be default. It is possible that

Network Service has been configured to use a proxy. If Security Manager reports credential

failures on the fleet when no passwords are present on the fleet, it is possible a proxy is

blocking the ability for Security Manager to query the pages it requires from the device to

determine if an Admin Password is set for the Embedded Web Server (EWS).

A utility named bitsadmin can be used to determine if a proxy is present on local service

accounts and can clear these settings. Try the following command to determine if network

Service is using a proxy:

Bitsadmin /util /getieproxy networkservice

Possible values include:

• NO_PROXY—Do not use a proxy server.

• AUTODETECT—Automatically detect the proxy settings.

• MANUAL_PROXY—Use an explicit proxy list and bypass list. Specify the proxy list and

bypass list immediately following the usage tag. For example, MANUAL_PROXY

proxy1,proxy2 NULL.

If the command returns MANUAL_PROXY or AUTODETECT, try setting Network Service to run

without a proxy by typing the following command:

Bitsadmin /util /setieproxy networkservice NO_PROXY

Splunk Dashboard for HPSM

It’s possible to use view task details in a dashboard.

Example:

For this a dashboard is provided “as is”. Using Splunk to analyze HPSM logs is possible, but keep in

mind that the HPSM support team is not supporting Splunk, nor the Splunk dashboard.

To use the dashboard in Splunk, follow the following steps:

1. In Splunk, click on Dashboard and Create New Dashboard.

2. Provide a title, like HPSM_TaskandSubtaks_Classic and click on Classic Dashboard and click

on Create.

3. Click on Source and remove all text.

Copy and paste all text from the Splunk dashboard text in Appendix E click on Save and Refresh

Note: Make sure you have enabled structured logs and debug mode for FMEA logging.

Installation Issues

Security Manager uses the Microsoft installer for installing and upgrading the software. If anything

goes wrong during installation, the logs file for the installation attempt may provide some clues.

For proper Security Manager installation and operation, specific Microsoft software must be

present. The requirements are listed in the install and setup guide.

If these are not present on the system, the installation process installs some of the required

software. The installer checks for the presence of IIS and attempts to enable it and necessary

configuration elements if not present. The installer also provides an option to install SQL Express if

desired or to use an existing SQL server location. It launches a series of SQL scripts to ensure the

database is at the current schema version to match the software version. Proper permissions need

to be present for the user running the upgrade on the SQL database for the installer to upgrade the

database.

Common SQL issues

By far the most common reason an installation goes awry is related to SQL issues. Whether using

local or remote SQL Server, Express or Full, the rules are essentially the same. In every case,

Security Manager needs access to a SQL server instance. It can either create a new database,

upgrade an existing database, or attach to an existing database, depending upon the situation and

the user rights. If Security Manager is instructed to install SQL Express on the local machine, a SQL

instance and database for Security Manager will be created by the Security Manager installer. If

Security Manager is pointed to an existing local or remote SQL server and instance during

installation, proper rights must be present for the user running the installation to be able to create

or update a SQL database wherever SQL server may reside. Proper rights must also exist on the

database itself for the user which the Security Manager service runs under to be able to read from

and write to the database.

TIP: When installing and upgrading Security Manager, by default the user permissions of the user

who is logged into the machine and running the installer executable will be used and must have

proper rights on the SQL server to either create a database or update an existing database. All the

installer does is run SQL scripts to create or alter a database, and naturally any user running those

commands needs to have proper SQL rights. In this case it is the Windows user who is running the

installer. During the installation it is possible to select a different windows user for

installing/upgrading the database.

Creating a database requires Create database rights on the SQL instance. Upgrading an existing

database requires DBO rights on the database. For normal operation of Security Manager after

installation, the user running the Security Manager service (default as Network Service) needs to

have permissions to at least read and write to the database, DBO rights preferred.

Each Security Manager installation must point to its own unique database, multiple installations

cannot share a database. The instance can be named or default, and the instance can have as many

databases as possible including a Web Jetadmin database. Several techniques are available to

allow Security Manager to install/use a SQL database in any location.

There are three scenarios where Security Manager will interact with Microsoft SQL:

• Creating a database during installation of Security Manager

• Upgrading a database during upgrading of Security Manager from one version to another

• Running Security Manager to manage security features on a fleet of devices

Each scenario requires a different set of SQL rights for potentially different users.

• Create Database – Windows user running the installer executable needs at minimum

Create Database rights on the SQL instance. SA rights would certainly work.

• Upgrade Database – Windows user running the installer executable to upgrade versions

needs DBO rights on the database to perform potential commands on the database such as

insert, update, alter, create table.

• Run Security Manager – the Windows account that runs the Security Manager service (default of

Network Service) needs DBO rights on the database to perform operations such as reading and

writing.

NOTE: The “HP Security - Using SQL Server” whitepaper explains how to run with less rights or a

different account if desired.

The first sign of trouble if something goes wrong might be an error when launching Security

Manager that indicates the SQL database cannot be opened.

In the following error, the message indicates that Windows Authentication is attempting to login the

user the Security Manager runs under (Network Service in this case) into the correct remote

server\instance name but is being rejected.

Use the following log file to find potential causes:

C:\Program Files (x86)\HP Security Manager\log\HPSM_Service.log

A successful login attempt will display the server\instance, database name, database version

and Security manager version:

This would happen if Network Service did not have DBO rights to use the database. Use SQL

Management Studio to confirm the user running the Security Manager service has DBO rights.

Also, make sure the Security Manager service was restarted after making the changes to the

database rights.

Ensure the HPSM_Service.exe.config file contains the correct entries:

• The server or instance name is not correct. Double-check the spelling of each.

• It is possible network related issues are preventing the connection to the remote instance.

Common troubleshooting steps include:

• Fully qualify the remote SQL server name in the configuration file if name resolution

issues are present or use the IP address instead of the hostname.

• TCP/IP must be enabled on the remote SQL server instance. Use SQL Server

Configuration Manager to confirm.

• Check firewall settings to ensure the port that is used for the remote connection is open.

The default port is 1433.

• SQL Server may default to using a dynamic port. Either configure to use a fixed port or

start the SQL Browser service to allow for remote connections.

• Use SQL Management Studio and/or Windows ODBC to connect to the remote SQL

server/instance from the same machine as Security Manager to at least prove a Windows

user account can access the server/instance from the Security Manager machine.

• If the HPSM_Service.log indicates table columns are missing, the database may not have

been upgraded due to insufficient rights by the user running the upgrade. This scenario is

described previously in this document with steps on how to uninstall/reinstall to rectify.

• The database tables should always begin with DBO as the schema i.e., DBO.DeviceTable.

2018-01-11 11:31:29,338 INFO Service [4] - TaskSupervisor.Init - HPSM Starting

2018-01-11 11:31:41,427 INFO Service [4] - Successfully started DSSessionVariables() -

hibernate session

2018-01-11 11:31:41,430 INFO Service [4] - ScheduleTaskManager.RetryDBConnection Testing

DB connection to: Server=(local)\EXP2014;initial catalog=HPIPSC;Integrated Security=SSPI;

2018-01-11 11:31:41,437 INFO Service [4] - - Done TestDBConnection

2018-01-11 11:31:41,437 INFO Service [4] - ScheduleTaskManager.RetryDBConnection Testing

DB connection successfull to: Server=(local)\EXP2014;initial catalog=HPIPSC;Integrated

Security=SSPI;

2018-01-11 11:31:41,593 INFO Service [4] - Service Starting up - Init()

2018-01-11 11:31:42,571 INFO Service [4] - Service Version:

3.1.0.65238

2018-01-11 11:31:42,594 INFO Service [4] - Shrinking the DB log file

If some of the tables begin with a Windows username as the schema, it is very likely that the user

who upgraded or created the database was a member of a Windows group when assigned SQL

rights.

The default schema for a user can be defined by using the DEFAULT_SCHEMA option of CREATE

USER or ALTER USER. If no default schema is defined for a user account, SQL Server will assume

DBO is the default schema.

IMPORTANT: If the user is authenticated by SQL Server as a member of a group in the Windows

operating system, no default schema will be associated with the user. If the user creates an object, a

new schema will be created and named the same as the user, and the object will be associated with

that user schema. The fix to this scenario is to either rename the schema in the affected tables to

indicate DBO, or better yet, assign a default schema of DBO to the Windows group to which the user

belongs.

Database Upgrade Failure

A very common scenario involves the installer completing the installation, but improper rights were

present on the account running the installer to upgrade the SQL database tables. When this

happens, you have a new Security Manager version trying to use older SQL tables. The log files are

filled with statements indication tables and/or columns are missing the database.

Two options exist in the case:

• Roll back to the previous version

• Attempt to repair the database to match the Security manager version

To roll back to the previous version, follow these steps:

1. Under Programs and Features, uninstall Security Manager and when asked if you would like

to delete the database, select No.

2. Finish the uninstall and when prompted, keep the license files intact.

3. Install the previous version, pointing to the remote server\instance where the database still

resides.

NOTE: Since the database was never upgraded when Security Manager was upgraded

earlier, it should still match the previous Security manager version and work fine with it.

4. Make sure the proper rights/permissions are in place before attempting the upgrade again.

Using SQL script to repair/upgrade/install the database

SQL scripts are also available that can repair the database and upgrade it to match the Security

manager version to update the database. This is certainly a much faster resolution to the problem.

The scripts are installed with HPSM (InstallSqlScripts.zip in the following directory:

C:\Program Files (x86)\HP Security Manager)

The scripts can be run by someone such as the DBA who has proper rights to upgrade the database.

The SQL scripts are run manually from a command prompt. They essentially mimic the SQL files the

installer runs when “install a new or update an existing database” is selected.

The script is launched by executing a batch file named InstallOrUPgradeRemoteDb.bat. The zip file

contains a Readme called Readme_InstallSQLscripts.txt with explanation and syntax of the SQL

scripts.

NOTE: Before HPSM 3.5 the script was called InstallDBrmt.bat and did not have an option to specify

the databasename. The syntax for the installDBrmt.bat script is explained in Appendix B.

When you run the SQL scripts from a command line, you must use the proper syntax It will not tell

you if your syntax is bad, but the log files that are created in the folder where you ran the scripts

will be full of errors if the syntax is bad.

If you use bad syntaxes, the command line will display as if it is creating tables. It is not. Log files

are created in the same folder containing the scripts. The logs will be filled with errors if

permissions are not correct or improper syntax was used.

The script first looks to see if the database is present. If not present, the script will create the

database if the user running the script has Create Database rights on the SQL instance. If an

existing database is present, the script walks through a routine to see what schema is present and

updates to the latest schema, if the user running the scripts has DBO rights on the database.

After the script completes, restart the HP Security Manager service. All should be well as the

software and database now match schema versions. See the whitepaper

HP Security Manager –

Using Microsoft SQL Server

for more information on running the SQL scripts.

If Security Manager still generates errors while attempting to launch or hangs indefinitely, the

database is probably still not upgraded. If you have a new Security Manager trying to use an old

database, statements display over and over in the service log indicating required tables are not

present:

2018-01-04 14:33:53,731 ERROR

NHibernate.Util.ADOExceptionReporter [4] - Invalid object name 'ScheduledReportsTable'.

In this example, ScheduledReportsTable is a new table only available in the 3.1 version of Security

Manager, and it is complaining it cannot find the table. That is very typical of new version using an

old database.

Here is a screen shot generated by viewing the database using SQL Management Server showing the

table that provides the database schema version. It should be version 7 for Security Manager 3.1.

The HPSM_Service.log file also indicates the schema version. Notice too how there is a

DBO.ScheduledReportsTable. That is another quick method to see if the database is upgraded or not.

Figure: Schema version indicated

Figure: Scheduled Reports Table listed

This can happen due to the following reasons:

• The account running the Security Manager installer does not have DBO rights on the database

• during an upgrade.

• The user chooses “connect to and existing database” instead of “install or upgrade an

existing database” during an upgrade attempt but the database was never upgraded

manually using the SQL scripts.

• The SQL scripts are run to manually upgrade the database but the account running the

scripts does not have DBO permissions on the database to upgrade it.

• The SQL scripts may be the broken ones included in the Security Manager folder of version 3.1.

• The SQL scripts are run using the wrong syntax on the command line.

Error-Maybe a DB access issue in HPSM_service.log file

A “Maybe a DB access issue” error is listed in the HPSM_service.log file:

Error,3/28/2021 4:31:41 AM,HP JetAdvantage Security Manager,0,None,"Error - Maybe a DB access issue

- calling SetRecoveryMode() System.Data.SqlClient.SqlException (0x80131904): This server is part of

Recovery Model exception list, Database Recovery Model should be in FULL for this server. If need any

clarification, Please reach out SQL Operation team Supervisor(s)

HPSM re-indexes during the nightly maintenance and changes the DB Recovery mode to SIMPLE

before indexing and then restores the previous setting after the re-indexing operation has been

completed. If the HPSM service does not have the rights to change the recovery model, then the error

will display.

This re-indexing can be disabled by setting the IndexPerformanceTuning to true in the

HPSM_service.exe.config file, followed by a restart of the HPSM service.

Problems Launching Security Manager Web Interface

First, check if the HPSM service is still running. When the HPSM service is running in debug mode

you can also check the HPSM_service.log file if the HPSM service can connect correctly to the

database. If the HPSM service is running and can connect to the database, then you need to

check the HPSM application pool.

Is the application pool still running after some time, or is it automatically getting stopped?

If it is getting stopped, check the HPSMWeb.log and event viewer for more information.

Web page not displayed while HPSM service is running and

connecting to DB

The following two subsections (bindings/firewall problem and browser setting causing issues)

describes issues when HPSM service can connect to the database.

Bindings/firewall problem

The Security Manager browser-based interface requires Internet Information Services (IIS) to

operate. The installer will verify that IIS is enabled with the proper settings enabled and will offer to

enable the proper settings if desired. The Installation Guide specifies the proper IIS setting to be

enabled to perform the configuration manually if preferred.

If the installer fails to set some of the IIS settings, it may be necessary to configure them manually.

Since the installer is attempting to enable IIS, it may prompt for a machine restart. You can use IIS

Manager under Administrative Tools to determine if the HPSM application pool and HPSM web site

are present and configured properly.

An easy test to determine if IIS is functioning properly is to see if you can browse to the default IIS

web page: http://localhost:80. If you can’t browse to the default page, it is likely you will not be

able to browse to the Security Manager page either.

When you still cannot access the web page, you might receive the following error message in your

browser when Security Manager is launched in a browser:

If the application pool is still running after attempting to reach the page, then there is a binding

problem of firewall.

To change this port, or if the port is being blocked, it can be changed by configuring the bindings for

the HPSM web site under IIS Manager. Expand Sites in the left pane, select HPSM, and then select

Bindings.

Figure: HPSM Binding settings

Make sure that the Type is set to https, Port to 7637 and that as an SSL certificate the HP Security

Manager Self signed self-certificate or a server certificate is selected. Change it to a different port

if it is blocked by the firewall or open the port in the firewall.

The self-signed certificate allows the data to be encrypted between client and server, while an

existing server certificate not only encrypts data but also provides trust that the server is who it

says it is. IIS will always search and bind for the server certificate in the personal store of computer

account. An identity certificate needs to be of the type “Server Authentication” to provide trust.

Issues have been seen in cases where there are multiple certificates to choose from with the same

name during installation, and if an improper certificate is chosen, the bindings will not be created for

the web site. Security Manager will not launch, and the HPSM web site will also contain a couple of

icons instead of the 20 or so it should contain. The easy fix to this problem is to select the correct

port for the bindings (7637). The icons will display, and Security Manager will run properly.

Browser settings causing issues

The new browser-based interface supports either Microsoft Internet Explorer (IE) or Google Chrome.

The following settings may need to be configured on certain machines or operating systems if

Security Manager is having difficulty loading and the application pool is running correctly.

If the login screen for Security Manager is not displaying, remove the Display intranet sites in

Compatibility View setting. To do this, click the ALT key to open the options in IE. Under Tools, select

Compatibility View, and then clear the Display intranet sites in Compatibility View option.

Internet Explorer may require the “Bypass proxy server for local addresses” box to be checked

under Internet Options, Connections, LAN Settings if the login screen for Security Manager is not

displaying.

Windows 10 may require HTTP2 to be disabled in the browser if Security Manager continually

logs out the user.

Web page not displayed and HPSM service is automatically stopping

In some situations, you might see that the HPSM service is stopping quickly after starting the

service. When trying to browse to the application you will see errors like Can’t reach this page.

This can have multiple reasons and the HPSM.service.log will have to be used to find out why this

happened.

ServerConfig could not initialized in HPSM_service.log

Entries like the following might be listed in the HPSM.service.log file:

022-01-27 02:01:40,807 INFO Service [4] - TaskSupervisor.Init - HPSM Starting

2022-01-27 02:01:52,229 INFO Service [4] - Successfully started DSSessionVariables() - hibernate

session

2022-01-27 02:01:52,244 INFO Service [4] - ScheduleTaskManager.RetryDBConnection Testing DB

connection to: Server=WWOED000SRV0327;initial catalog=HPSECMGR;Integrated Security=SSPI

2022-01-27 02:01:52,244 INFO Service [4] - - Done TestDBConnection

2022-01-27 02:01:52,244 INFO Service [4] - ScheduleTaskManager.RetryDBConnection Testing DB

connection successful to: Server=WWOED000SRV0327;initial catalog=HPSECMGR;Integrated

Security=SSPI

2022-01-27 02:01:52,244 INFO Service [4] - Service Starting up - Init

2022-01-27 02:01:52,369 INFO Service [4] - BizLogicMgr.IsValidProductDB - productId 6F1BC5E1-A0C9-

4CEF-B065-BBF5F487F289 metaData.ID6F1BC5E1-A0C9-4CEF-B065-BBF5F487F289

2022-01-27 02:01:52,526 ERROR Service [4] - BizLogicMgr.ValidateServerCertificate Certificate error:

{0}RemoteCertificateChainErrors

2022-01-27 02:01:52,526 ERROR Service [4] - BizLogicMgr.Init - Error ServerConfig not initialized -

Unknown error: System.Security.SecurityException: The source was not found, but some or all event

logs could not be searched. To create the source, you need permission to read all event logs to make

sure that the new source name is unique. Inaccessible logs: Security, State.__ at

System.Diagnostics.EventLog.FindSourceRegistration(String source, String machineName, Boolean

readOnly, Boolean wantToCreate)__ at System.Diagnostics.EventLog.SourceExists(String source,

String machineName, Boolean wantToCreate)__ at

System.Diagnostics.EventLogInternal.VerifyAndCreateSource(String sourceName, String

currentMachineName)__ at System.Diagnostics.EventLogInternal.WriteEntry(String message,

EventLogEntryType type, Int32 eventID, Int16 category, Byte[] rawData)__ at

System.Diagnostics.EventLog.WriteEntry(String source, String message, EventLogEntryType type, Int32

eventID, Int16 category, Byte[] rawData)__ at Common.DataConstants.SetURLSetting(String

settingName, String defaultValue, String& memberSetting)__ at

Common.DataConstants.InitializeFromAppConfig()__ at

LocksmithBusinessLogic.BizLogicMgr.Init(Boolean& startSSLListner)__The Zone of the assembly that

failed was:__MyComputer

2022-01-27 02:01:52,526 ERROR Service [4] - HP Security Manager: Unexpected Error Initializing -

Shuting down: System.Security.SecurityException: The source was not found, but some or all event logs

could not be searched. To create the source, you need permission to read all event logs to make sure

that the new source name is unique. Inaccessible logs: Security, State.__ at

System.Diagnostics.EventLog.FindSourceRegistration(String source, String machineName, Boolean

readOnly, Boolean wantToCreate)__ at System.Diagnostics.EventLog.SourceExists(String source,

String machineName, Boolean wantToCreate)__ at

System.Diagnostics.EventLogInternal.VerifyAndCreateSource(String sourceName, String

currentMachineName)__ at System.Diagnostics.EventLogInternal.WriteEntry(String message,

EventLogEntryType type, Int32 eventID, Int16 category, Byte[] rawData)__ at

System.Diagnostics.EventLog.WriteEntry(String source, String message, EventLogEntryType type, Int32

eventID, Int16 category, Byte[] rawData)__ at Common.DataConstants.SetURLSetting(String

settingName, String defaultValue, String& memberSetting)__ at

Common.DataConstants.InitializeFromAppConfig()__ at

LocksmithBusinessLogic.BizLogicMgr.Init(Boolean& startSSLListner)__The Zone of the assembly that

failed was:__MyComputer

In this case the event source “HP Security Manager” which should become visible in the event viewer

for HPSM related events can not be created/read.

Example of HPSM event message in the event viewer:

First, check if the account (by default Network service) running the HPSM service as at least

read permissions to the corresponding registry key.

1. Open the Registry Editor:

2. Select Start then Run

3. Enter regedt32 or regedit

4. Navigate/expand to the following key:

5. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\Security

6. Right-click the entry and select Permissions

7. Add the Network Service user (or the account running the HPSM service)

8. Give it Read permission

9. Restart HPSM service.

If the correct permissions were already available or if the issue was not resolved, try to run HPSM

temporarily as Local System.

1. Go to services

2. Right-click the HP Security Manager service and select Properties

3. Click on the Log On tab and select the radio button Local System Account

4. Restart HPSM service.

The event source HP Security Manager is created with the Local System Account.

After HPSM has been running for a few minutes, verify in the HPSM_service.log file that the

error “Server config could not be initialized” is no longer occurring. Change HPSM service

back to the desired account.

Invalid length for a Base-64 char array or string in HPSM_service.log

When the HPSM service is stopping automatically after trying to connect to the DB, you might see the

following in the HPSM_service.log:

2023-12-14 12:23:47,771 DEBUG Service [4] - ScheduleTaskManager.DispatchEvents Initializing

2023-12-14 12:23:47,771 INFO Service [4] - Service Starting up - Init

2023-12-14 12:23:47,771 DEBUG Service [4] - BizLogicMgr - GetMetaData

2023-12-14 12:23:47,966 INFO Service [4] - BizLogicMgr.IsValidProductDB - productId 11870252-D338-

4BCD-9EBD-8A77C043D65D metaData.ID11870252-D338-4BCD-9EBD-8A77C043D65D

2023-12-14 12:23:48,193 DEBUG Service [4] - DecryptUsingDPAPI method called.

2023-12-14 12:23:48,193 DEBUG Service [4] - Error - DecryptUsingDPAPI: Invalid length for a Base-64 char

array or string.

This happens when the hpsm_service.exe.config has been configured with an sql user account to access the

database and the sql password has been encrypted by HPSM during the first successful start of the hpsm

service.

Example of encrypted sql user password in hpsm_service.exe.config file:

The web.config file also needs to be configured to use the SQL user account to connect to the DB and the

password should have been encrypted by HPSM in the web.config file after the password has been encrypted

in the hpsm_service.exe.config file.

If the sql password is not encrypted in the hpsm_service.exe.config and web.config on all locations where the

database connection is specified, then you’ll run in the invalid length error.

Solution: stop hpsm service and hpsm application pool. Re-edit the hpsm_service.exe.config and web.config

file and make sure that the unencrypted sql password is specified.

Now start hpsm application pool

Finally start hpsm security manager service

Issues when running HPSM Application Pool

As network service - Server Error in “/” Application, Cannot open database

The Security Manager service and the HPSM application pool must have the proper permissions to

access the Security Manager service database.

If the service and database are installed on the same computer, the installation process manages

the assignment of database permissions by assigning Network Service to run both the HPSM

service and the HPSM application pool.

If the service and the database are installed on separate computers, you must configure the correct

permissions for the remote database, otherwise the HPSM service cannot login to the database and

you might receive an error like the following:

If the HPSM service has been configured to run under a user or service account which has access to

the database, you still might receive a server error, due to login problems of the HPSM application

pool.

By default, the HPSM application pool will be configured with NetworkService, but this can be

changed if desired to run under the same account as the HPSM service or by providing access to

the database for the Network Service.

Open IIS, right-click the HPSM application pool, select Advanced Settings, click on … , select Custom

Account, specify the account which should be used, and then select OK.

See the whitepaper titled “HP Security Manager - Using Microsoft® SQL Server” for more information

about the required rights for a remote database. It is also possible that some IIS settings are not

configured. Double check that all of the IIS settings were enabled as described in the “HP Security

Manager Installation Guide“.

If the browser displays an error that it cannot read the web.config file, search the internet for more

information on the error code.

32-Bit Applications not enabled for HPSM application pool

The Enable 32-bit Applications setting for the HPSM application pool may need to be toggled. To do

this, follow these steps:

1. In IIS, select Application Pools, right-click HPSM, and then select Advanced Settings (or select

Advanced Settings in the right -pane).

The following screen displays:

2. Change the setting for Enable 32-bit Applications from False to True.

HTTP 500.19 Internal Server Error with config source reference to X-Frame-

Options

When running HPSM Application Pool on some systems, you might receive an HTTP 500.19 error with

config source reference to X-Frame-Options.

Figure: HTTP 500.19 error received

The Config Source shows in red information about X-FRAME-OPTIONS in line 338. During installation

of HPSM, the installer configures the IIS settings for HPSM. It updated the applicationHost config file

which is by default in the following location:

C:\Windows\System32\inetsrv\Config

For HPSM, the customHeaders section for the httpProtocol should be configured as the following:

</customHeaders>

</redirectHeaders>

</httpProtocol>

Other customHeaders content should be removed. For example:

Before making any changes save a copy of the file and store it in a save location. After that

remove, all special customHeaders add name sections, restart the HPSM application pool

and restart the HP Security Manager service.

HTTP 500.19 Internal Server Error with error code 0x800700c1

When running HPSM Application Pool on some systems, you might receive an HTTP 500.19 error with

error code 0x8000700c1.

Figure: HTTP Error 500.19 - Internal Server Error Error Code 0x800700c1.

Potential fix (it is possible that other IIS configuration settings are causing the same issue):

1. Open IIS (Internet Information Server).

2. Click on server module node at the top of the left navigation pane, select Modules, and then select

Open Feature or double click Modules.

3. Right-click DynamicCompressionModule and select Unlock.

4. Right-click StaticCompressionModule and select Unlock.

5. Expand Sites and select HPSM.

6. In the right pane in the IIS section , select "Modules" and select “Open Feature”, or double click the

Modules option.

Figure: Modules option highlighted on HPSM Home screen

7. Right-click DynamicCompressionModules and select Remove.

8. Right-click StaticCompressionModule and select Remove.

9. Restart the IIS.

As a non-admin user or service account (error 503)

When running HPSM Application Pool, if the service account is NOT a member of the local

administrator group on the Security Manager server, additional steps are required to ensure the

service account can access the service control manager like the Network Service can.

If these additional steps are not taken, the HPSM application pool will stop as soon as you try to open

the HPSM web page, and a 503 error will likely be seen when attempting to launch Security Manager.

The HPSMWeb.log in the directory

C:\Program Files (x86)\HP Security Manager\WebApp\log

and the

Windows Event Viewer will contain errors indicating there are not sufficient rights to access service

control manager:

<Data>HP JetAdvantage Security Manager: Unexpected Error Initializing - Shuting down:

System.InvalidOperationException: Cannot open Service Control Manager on computer '.'. This operation

might require other privileges. ---> System.ComponentModel.Win32Exception: Access is denied --- End of

inner exception stack trace --- at

System.ServiceProcess.ServiceController.GetDataBaseHandleWithAccess(String machineName, Int32

serviceControlManaqerAccess) at

System.ServiceProcess.ServiceController.GetDataBaseHandleWithEnumerateAccess(String

machineName) at System.ServiceProcess.ServiceController.GetServicesOfType(String machineName,

Int32 serviceType) at System.ServiceProcess.ServiceController.GetServices() at

LocksmithBusinessLogic.BizLogicMgr.ValidateProductAndDB() at

LocksmithBusinessLogic.BizLogicMgr.Init()</Data>

By default, non-administrators cannot access the service control manager and cannot stop/start

services.

To assign the rights needed for a non-admin account to access the service configuration manager,

run the following from command prompt then restart the Security Manager service. This command

provided privileges so that authenticated users can run the service control manager as required by

Security Manager.

Open a command prompt as administrator, browse to

C:\Windows\System32

, and type the following

command:

sc sdset SCMANAGER

D:(A;;CCLCRPRC;;;AU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)S:(AU;FA;KA;;;WD)(A

U;OIIOFA;GA;;;WD)

If successful, the line ‘[SC] SetServiceObjectSecurity SUCCESS’ is displayed.

The final step is to make sure the service account is a member of the local IIS_USRS group.

Restart the Security Manager service, recycle the HPSM application pool, and you should be able to

successfully open Security Manager.

When the HPSM application pool is running and the UI is launched, the process w3wp.exe will start.

For each application pool inside IIS another w3wp.exe process will start.

When you add the column Command line in task manager you can see which process is used by which

application pool.

In the command line column, you will see for the w3wp.exe process like:

C:\windows\SysWOW64\inetsrv\w3wp.exe

-ap “HPSM” -v “v4.0” -l “webengine4.dll” -a

\\.a\pipe\iisipm324143214-2432424-sdfasfds- h

“C:\inetpub\temp\apppools\HPSM\HPSM.config” -w – m 0 -t 20 -ta 0

For HPSM this also means that full access is required to the following file/path:

C:\Inetpub\temp\apppools\HPSM\HPSM.config.

If the HPSM application pools is still crashing after trying to access the UI when running the HPSM

application pool as a different user/service, then the account is most likely too restrictive. In that

case try to run the default application of IIS as the same user/service. If the default application pool is

also crashing when you try to access the default IIS homepage (http://localhost:80), then it confirms

that the problem is outside of HPSM and caused by a too restrictive user or service account.

To resolve this issue, add the user/service account to the local administrators group.

Access Denied when trying to logon to HPSM

Users, even (local) Administrators might get an Access Denied when trying to logon to Security

Manager.

Figure: Access denied error when trying to log in

Only users that are members of the HPIPSC group will have full access to the HP Security Manager

and the specific user will have to be added to this group first.

Licensing Issues

Licensing Issues in multi-homed environment: Failed to connect to

licensing server

Issue

In a multi-homed server (several NICs installed). It is possible that the HPSM service tries to reach

the HP Print License service on the wrong IP address/NIC with the result: Failed to connect to

licensing server in the Dashboard screen and unable to add new licenses.

Perform the following steps to check if this is happening:

1. Retrieve the table DBO.ServerConfigTable (this contains one entry) and check for the

LicenseServerAddress in this table.

2. Retrieve the table DBO.ServerTable and check which IP addresses are listed.

If the second IP address in this table is used for communication with all the devices, then HPSM will

not be able to communicate to the licensing server.

Solution

1. For every NIC , open the Control Panel, Network and Sharing Center and select Change Adapter

Settings.

2. Right-click every Adapter/NIC.

3. Select Internet Protocol Version 4 (TCP/IPv4).

4. Select Properties.

5. Select Advanced and disable Automatic metric.

6. Add a value for the Interface metric.

7. The interface which is used for communication with the devices should have the lowest number as

lower number indicates higher priority.

8. Now restart the HP Security Manager service. During the service startup, HPSM will update the

priority order in the DBO.ServerTable and update the DBO.ServerConfigTable accordingly.

NOTE: Changing the adapter order can also done in other ways, see for example:

https://www.windowscentral.com/how-change-priority-order-network-adapters-windows-10

Licensing does not work when HPSM’s hostname is changed after

installation

If in the rare case the Security Manager server has obtained a new hostname, Flexera likely needs

some help to reflect that change. Flexera refers to some keys in the windows registry related to

server hostname.

The following error message in the HPSM Service log indicates an error while HPSM service tries to

communicate with the Flexera Service. The line containing “GetLastFlexeraError" typically provides

error information.

2021-05-25 13:54:13,478 ERROR Service [License Task 5/25/21 01:54] - GetFeatureNames() - Unable

to get any features from Flexera. Retry 2

2021-05-25 13:54:14,494 ERROR Service [License Task 5/25/21 01:54] - GetLicenseFeatures() -

Exception - Attempted to read or write protected memory. This is often an indication that other

memory is corrupt. - Stacktrace - at

HP.Print.License.CSharpWrapper.NativeMethods.HPGetLicenseFeaturelist(Int16 jobType, Int32

flag)__ at HP.Print.License.CSharpWrapper.Communication.GetLicenseFeatures(JobInfo job, Int32

searchFirst) in C:\Code\LS352\CSharpWrapper\Communication.cs:line 64

2021-05-25 13:54:14,494 ERROR Service [License Task 5/25/21 01:54] - GetLastFlexeraError() - error

string is - GetLastFlexeraError - Error string is - No socket connection to license server

manager._Feature: IPSC_CMPS_License path: 27000@WIN-TEST;_FLEXnet Licensing error:-

7,10001_For further information, refer to the FLEXnet Licensing documentation,_available at

"www.flexerasoftware.com". - GetLastFlexeraError End

The error message “No socket connection to license server manager” indicates the following

possibilities:

• Flexera service not running.

• HPSM Service’s connection to Flexera service is blocked by a firewall.

• The registry entries telling HPSM Service where to connect to Flexera is not accessible, not

present, corrupted, or incorrect (example change of hostname).

One option is to uninstall Security Manager, keep database and licenses intact, then reinstall

Security Manager and point to the existing database. This would likely update the registry entries to

the new hostname.

Another option involves editing the registry entries manually and restarting the services rather than

a full uninstall/reinstall. The steps for this in HPSM 3.5 and older are different from the steps for

HPSM 3.6 and newer.

For HPSM 3.6 and newer

By default, HPSM 3.6 uses environment system variables for the variable HPQ_LICENSE_FILE

instead of registry entries. The system variable also contains multiple values for IPv4 and IPv6

usage with the following value:

27000@<computername>,27[email protected].1,27000@[::1]

Where <Computername> is the (host)name of the computer on which HPSM is running.

Figure: System variable HPQ_LICENSE_FILE

When the HPSM hostname has been changed, the values for the HPQ_LICENSE_FILE system

variable need to be changed accordingly. After this has been changed restart the HP Security

Manager Service.

For HPSM 3.5 and older

1. Stop the following services in this order:

• HP JetAdvantage Security Manager Service

• HP Print License Service

• Flexera Licensing Service

2. Open Registry Editor.

3. Find all the ‘HPQ_LICENSE_FILE’ keys in the registry and replace the values to reflect the new

hostname (they will all have the old hostname referenced after the @ sign). There will be

several keys in the registry for each machine and user account.

Figure: Example of one of the HPQ_LICENSE_FILE entries in the registry

Delete license files under the following directory (do not delete DemoLicense

file_ForStartup.lic).

C:\ProgramData\HP\HP Print License Service\LicenseFiles

Delete All files under RecoveryXXX directory (do not delete the directory itself).

C:\ProgramData\HP\HP Print License Service\RecoveryFile1 C:\ProgramData\HP\HP Print

License Service\RecoveryFile2

6. Restart the services in the reverse order that you stopped them:

• Flexera Licensing Service

• HP Print License Service

• HP JetAdvantage Security Manager Service

7. Check whether the HPSM server is working well with the demo license.

8. Install the issued license again.

'Insufficient licenses' error after nightly maintenance task with HPSM

3.6 or newer

If the message ‘insufficient’ licenses displays after the automatic nightly maintenance from HPSM,

then do the following:

1. Remove the current system variable HPQ_LICENSE_FILE

2. Stop the following services in this order:

• HP Security Manager Service

• Flexera Licensing Service

3. Open Registry Editor and look for the following registry key:

[HKEY_LOCAL_MACHINE\SOFTWARE\FLEXlm License Manager]

4. Create a new String Value with the name HPQ_LICENSE_FILE and the value where

<Computername> is the (host)name of the computer on which HPSM is running:

27000@<computername>,2[email protected].0.1,27000@[::1]

Figure: Computer is listed as the host name

5. A similar entry must be made in the registry key for the user running the HPSM service. By

default, this is the NETWORK service which has an SID of S-1-5-20. Look for the following

registry key:

[HKEY_USERS\S-1-5-20\Software\FLEXlm License Manager]

6. Create a new String Value under this key, with the string name HPQ_LICENSE_FILE and value

where <Computername> is the (host)name of the computer on which HPSM is running:

27000@<computername>,2[email protected].0.1,27000@[::1]

Figure: HPQ_LICENSE_FILE listed under FLEXlm License Manager registry key

7. If HPSM service is configured to run under other user privileges, that’s specific user’s entry

under HKEY_USERS must be modified.

8. Restart the services in the reverse order that you stopped them:

• Flexera Licensing Service

• HP Security Manager Service

9. Check whether the HPSM server is working well with the licenses.

If the issue persists, reboot the server, and check again.

'Insufficient licenses' after upgrading to HPSM 3.10 or later

From HPSM 3.10 onwards, HPSM blocks the usage of all licenses when a duplicate license is

detected in the folder: C:\ProgramData\HP\HPSM\LicenseFiles

Under Settings, Licenses zero licenses are displayed. When clicking on View License Details, no

information is displayed:

Solution:

1. Search CheckLicenseDuplication in the HPSM_service.log file to find which license file is duplicated

(highlighted in yellow)

2024-04-09 14:06:02,016 INFO Service [License Task 4/9/24 02:06] -

Existing License in Folder: "00AE27A81D7E0E551A96F6EB738B_

01000D9999CA5829DE54768B9764835E"

2024-04-09 14:06:02,016 INFO Service [License Task 4/9/24 02:06] -

Existing License in Folder: "0010FF9F8CB6C2CC792D73ADACCC_

9A00643AD80B01D7E7729D8E5DAFECD4"

2024-04-09 14:06:02,016 INFO Service [License Task 4/9/24 02:06] -

Existing License in Folder: "00C409C1D64336C029596ED26BD8_

AF00E2F416EBEBA935E52D14275D2C3E"

2024-04-09 14:06:02,016 ERROR Service [License Task 4/9/24 02:06] -

LicenseWrapper.CheckLicenseDuplication: License :

"00EFEFA43EC385D06FD98DA1EF4B_ 41008270CC7DB804E96281BE66F20B1D"

2024-04-09 14:06:02,016 DEBUG Service [License Task 4/9/24 02:06] -

LicenseMgr.CheckOut

2024-04-09 14:06:02,016 DEBUG Service [License Task 4/9/24 02:06] -

LicenseMgr.CheckOut #: 0

2024-04-09 14:06:02,016 DEBUG Service [License Task 4/9/24 02:06] -

BizLogicMgrHelper - GetDataSetCount returned: 9

2024-04-09 14:06:02,016 DEBUG Service [License Task 4/9/24 02:06] -

LicenseWrapper.CheckOut for feature: total # required: 9

2024-04-09 14:06:02,016 ERROR Service [License Task 4/9/24 02:06] -

CheckOut() - Licenses Available - 0 - Seats Missing - 9 - Licenses

Assigned -0

2024-04-09 14:06:02,016 DEBUG Service [License Task 4/9/24 02:06] -

LicenseWrapper.CheckOut numSeatsMissing: 9

2024-04-09 14:06:02,016 DEBUG Service [License Task 4/9/24 02:06] -

LicenseMgr.UpdateCountAndStatus

2024-04-09 14:06:02,016 DEBUG Service [License Task 4/9/24 02:06] -

BizLogicMgr - GetServerConfig

2024-04-09 14:06:02,031 DEBUG Service [License Task 4/9/24 02:06] -

Method : DataProtection.Encrypt. Enter

2024-04-09 14:06:02,031 DEBUG Service [License Task 4/9/24 02:06] -

License data: -Feature: -Expires in: 0 -# Assigned: 0 -# Available: 0

-# Total: 0 -Connection status ok: False -When: -Feature: -Feature:

12/31/99 11:59

2. Use the first number, in this case 00EFEFA43EC385D06FD98DA1EF4B

3. Add spaces to this after every 4

character, thus now it becomes

00EF EFA4 3EC3 85D0 6FD9 8DA1 EF4B

4. Now search for this value inside the license files in the directory:

C:\ProgramData\HP\HPSM\LicenseFiles

You should find the same value in multiple license files:

5. Move the duplicate license files to a backup location.

6. Stop HPSM service

7. Stop and restart flexera service

8. Restart HPSM service.

Flexera service Will Not Start (Flexera started and then stopped)

Shortly after starting the Flexera Service, you might see the following error message:

This means that the account running the Flexera service does not have Full Control of the HPSM

installation directory.

Solution

Go to the HPSM installation directory (Program Files (x86)\HP Security Manager) and provide Full

Control to this directory for the account running the Flexera service.

‘Failed to retrieve the data’ error in the HPSM dashboard and zero

licenses available

Whenever the error message “Failed to retrieve the data” in HPSM dashboard you need to check the

HPSMWeb.log file in the directory \Program Files (x86)\HP Security Manager\WebApp\log for

ERRORs. The following error might be listed in the log file:

020-11-16 13:45:04,560 INFO Web [71] - et=W2, controller=Licenses, action=Get,

httpresponsecode=InternalServerError, elapsed=60.525

2020-11-16 13:45:04,560 ERROR Web System.ServiceModel.CommunicationObjectFaultedException:

The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for

communication because it is in the Faulted state.

Server stack trace:

at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)

at System.ServiceModel.Channels.CommunicationObject.Close()

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage

retMsg)

The Flexera service might need more than 5 minutes to load all license files (if there is a high number

of license files). This might be caused by a WCF timeout. By default this is set to 5 minutes.

Solution 1:

1. Stop HPSM service

2. Restart the hpsm application pool in IIS (internet information services).

3. Start HPSM service

Solution 2:

Step 1. Increased the clientTCPBindingOpenTimeout, clientTCPBindingSendTimeout,

clientTCPBindingReceiveTimeout and clientTCPBindingCloseTimeout settings in the Web.config file

and the HPSM_service.exe.config files to a higher value (for example 10 minutes for testing).

<!--ClientTCPBindingOpenTimeout : Increase the WCF timeout value to

open the WCF connection -->

<!--ClientTCPBindingSendTimeout : Increase the WCF timeout value

to send data in the WCF connection-->

<!--ClientTCPBindingReceiveTimeout : Increase the WCF timeout

value to receive data in WCF connection-->

<!--ClientTCPBindingCloseTimeout : Increase the WCF timeout

value to close the WCF connection-->

NOTE: The following value requirements must be met:

• All values must have the same timeout

• The same values must be used in both Web.config and HPSM_service.config files

• Increasing the timeout causes UI to wait longer up-to the specific timeout, so it may be

“busy,” until the request is completed or times out. So ideally, this should only be used if

licensed features in UI, like creating blank policy is affected.

Step 2: Restart the HPSM application pool and then restart the HPSM service.

Insufficient Licenses in HPSM dashboard and zero licenses

available with description of starting order of Flexera

components

Check for the following error in the HPSMService.log

2024-04-11 16:40:01,123 ERROR Service [License Task 4/11/24 04:39] -

GetLicenseFeatures() - Exception - Attempted to read or write protected

memory. This is often an indication that other memory is corrupt. -

Stacktrace - at

HP.Print.License.CSharpWrapper.NativeMethods.HPGetLicenseFeaturelist(Int1

6 jobType, Int32 flag)__ at

HP.Print.License.CSharpWrapper.Communication.GetLicenseFeatures(JobInfo

job, Int32 searchFirst)

2024-04-11 16:40:01,123 ERROR Service [License Task 4/11/24 04:39] -

GetLastFlexeraError() - Fetch last error message from Flexera -

GetLastFlexeraError - Error string is - No socket connection to license

server manager._License path: 27000@WIN-AVNRO0BGK9B, 27[email protected],

27000@[::1];_FlexNet Licensing error:-7,96 - GetLastFlexeraError End

2024-04-11 16:39:59,842 ERROR Service [4] - GetFeatureNames() - Unable

to get any features from Flexera. Retry 3

The retry count will keep increasing. At the same time the Flexera.log might be ending with:

16:37:49 (lmgrd) pid 4296

16:37:49 (lmgrd) SLOG: Summary LOG statistics is enabled.

16:37:49 (lmgrd) Detecting other license server manager (lmgrd)

processes...

In the task manager you can see in the Details tab, that lmgrd.exe is listed twice:

This means that the Flexera service has not loaded all license information yet. Once all licenses have

been loaded you will also see this loading process in the Flexera.log:

16:44:02 (lmgrd) License file(s):

C:\ProgramData\HP\HPSM\LicenseFiles\1.HP I-P Security_24260538_64.lic

C:\ProgramData\HP\HPSM\LicenseFiles\10.HP I-P Security_24261702.lic

C:\ProgramData\HP\HPSM\LicenseFiles\11.HP I-P Security_24261704.lic

C:\ProgramData\HP\HPSM\LicenseFiles\12.HP I-P Security_24261706.lic

C:\ProgramData\HP\HPSM\LicenseFiles\_DemoLicense_ForStartup.lic

HPSM will only be able to get the license count when the process HPQ.exe has been started.

This should be visible in the task manager and in the Flexera.log:

17:05:01 (lmgrd) Started HPQ (pid 4880)

After HPQ has been started, click on another tab in HPSM and go back to the dashboard. The licenses

should now be visible in the dashboard.

Mixing licenses limitations

Purchased licenses can be stacked, but mixing purchased with custom trial/evaluation licenses will

usually generate errors. Examples include the following:

• Attempt to add a permanent license for 8000 devices with no expiration tied to mac address

on top of a custom trial license that works on any machine: Failure - Cannot mix AnyHost and

Mac Address hosts for IPSC_DEVICES.

• Attempt to add a permanent license for 8000 devices with no expiration tied to mac address

over top of the demo license included in product: Success - IPSC_DEVICES replaces

IPSC_CMPS.

• Attempt to add a permanent license for 8000 devices with no expiration tied to mac address

over top of the downloaded 60-day trial license from the kiosk: Success - IPSC_DEVICES

replaces IPSC_DEMO.

• Install downloaded 60-day trial license available on kiosk on top of demo license included in

product: Success - IPSC_DEMO replaces IPSC_CMPS.

• Attempt to add another downloaded 60-day trial license from kiosk: Failure – cannot stack

multiple IPSC_DEMO licenses.

• Attempt to add a custom trial license for 100 devices expiring Apr. 30 on top of default demo

license included n product: Success, IPSC_DEVICES replaces IPSC_CMPS.

• Attempt to add another custom trial license for 550 devices expiring Aug 31 on top of another

trial license: Success, added to existing 100 trial, but days remaining reflects first file read.

Expired license is ignored. IPSC_DEVICES can stack together as long as all AnyHost.

Other Licensing Issues before HPSM 3.7

There are a couple of symptoms that would indicate something went wrong when applying the

licenses. The License Server Status indicates Error instead of Success:

This may be accompanied by an error at the bottom of the page indicating “Error: Failed to connect to

the license server.”

Another sign of a problem occurs when attempting to Add Licenses Now, a failure screen displays

indicating no licenses were added.

Here are some steps that can be followed to attempt to resolve these licensing issues:

1. Security Manager requires the proper startup of these 3 services.

• Flexera Licensing Service

• HP Print License Service

• HP JetAdvantage Security Manager

2. You may need to stop and restart these services. Stop them in this order:

• HP JetAdvantage Security Manager

• HP Print License Service

• Flexera Licensing Service

Restart them in the reverse order:

• Flexera Licensing Service

• HP Print License Service

• HP JetAdvantage Security Manager

3. Launch the UI.

The licensing process in Security Manager is handled by two services: Flexera and HP Print

License service. The latter relies on the former working.

There may be cases where the Flexera service cannot start because of reduced permissions on

the service account it is using. Windows Event Viewer may indicate the service cannot start

because of reduced permissions.

By default, the Flexera service runs under the Local Service account. If permissions are reduced

on this account and cannot be increased, try running the service under another account with

more permissions.

The services may also be starting up too slowly so that the later services cannot determine the

initial services are started.

Try manually starting the services with a time break of perhaps 30 seconds in between. If this

resolves the issue, a delay can be defined for the services when they automatically start.

Method one: Uninstall Security Manager

1. Make sure the uninstaller removes licenses.

2. Reinstall and add licenses.

3. When the uninstaller asks if you want to delete the database also, you can choose not to do so

if you prefer to maintain all the devices, policies, and remediation data from the previous

install.

4. Re-install Security Manager, point to the existing database, and then install the licenses.

Method two: Manually delete evaluation licenses

If you do not want to uninstall Security Manager, another technique for removing evaluation

licenses is to manually delete them.

To do this, follow these steps:

1. Stop the services in this order:

• HP JetAdvantage Security Manager service

• HP Print License service

• Flexera service

2. Delete the evaluation license file (do not delete the DemoLicense file) under:

C:\ProgramData\HP\HP Print License Service\Licenses

3. Delete all recovery files (LSRecovery.xml) under the recovery directories (if present):

C:\ProgramData\HP\HP Print License Service\RecoveryFile1

C:\ProgramData\HP\HP Print License Service\RecoveryFile2

4. Reboot the HPSM server (in most cases restarting first Flexera service, then the Print License

service and finally HP JetAdvantage Security Manager service, will not fix the licensing issue, a

reboot is required in most situations).

If both services are running and issues still arise, there may license type conflicts such as trying to

add purchased licenses alongside custom trial\evaluation licenses. Each license has unique

parameters such as machine type, expiration date, and feature type. Some varieties cannot be

mixed.

If problems still exist, there might be a port conflict. For example, Port 8888 needs to be open for

the HP Print License service so check firewall and/or McAfee type firewall products. It might be

easier to allow applications through the firewall instead of specific ports. Allow the following

through Rules:

C:\Program Files (x86)\ HP JetAdvantage Security Manager\HP Print License Service\lmgrd.exe

C:\Program Files (x86)\ HP JetAdvantage Security Manager\HP Print License Service\HPQ.exe

C:\Program Files (x86)\ HP JetAdvantage Security Manager\HP Print License

Service\HP.Print.License.Host.WindowsService.exe

Even though Flexera and the HP Print License service are on the same machine, other products

that use the HP Print License service may not have Flexera on the same machine. The HP Print

License Service requires the dedication of this TCP Port and cannot be modified for alternative

port assignment.

A successful use of port 8888 displays in the HPSM_Service.log file as follows:

2018-01-11 11:31:49,416 INFO Service [License Task 1/11/18 11:31] - LicenseWrapper.Connect using

IPs : 15.25.250.161 : 15.25.250.161 device features ::devices version: :host addr:

net.tcp://15.25.250.161:8888/LicensingService ok to LogErrors: True

An unsuccessful use of port 8888 displays in the HPSM_Service.log file as follows:

2015-04-07 11:15:27,146 ERROR Service - [License Task 4/7/15 11:15] - LicenseWrapper.Heartbeat -

(skipping) unexpected ERROR: System.ServiceModel.EndpointNotFoundException: Could not

connect to net.tcp://192.168.181.1:8888/LicensingService. The connection attempt lasted for a time

span of 00:00:21.0285341. TCP error code 10060: A connection attempt failed because the

connected party did not properly respond after a period, or established connection failed because

connected host has failed to respond 192.168.181.1:8888.

Other applications such as ePrint also use this HP Print License service but can use a different

version that conflicts with the version used by Security Manager. For this reason, ePrint and

Security Manager cannot coexist on the same machine. You may have to uninstall the existing HP

Print License Service under Programs and Features or Add/Remove Programs and reinstall Security

Manager to obtain the proper HP Print License Service.

Log files can indicate possible causes of the problem. Two log files exist for the license service:

C:\ProgramData\HP\HP Print License Service Files\Flexera.log C:\ProgramData\HP\HP

Print License Service Files\HPPLS.log

The Security Manager service log can also provide valuable information:

C:\Program Files (x86)\ HP JetAdvantage Security Manager\log\HPHPSM_Service.log

For example:

2015-04-07 11:15:27,146 ERROR Service - [License Task 4/7/15 11:15] - LicenseWrapper.Heartbeat -

(skipping) unexpected ERROR: System.ServiceModel.EndpointNotFoundException: Could not

connect to net.tcp://192.168.181.1:8888/LicensingService. The connection attempt lasted for a time

span of 00:00:21.0285341. TCP error code 10060: A connection attempt failed because the

connected party did not properly respond after a period, or established connection failed because

connected host has failed to respond 192.168.181.1:8888.

A “Flexera call completed” message in the HPPLS log file indicates the issue is not related to the

Flexera service.

A Winsock error usually means the service is having trouble opening a TCP socket. This might be

network or firewall related, perhaps caused by a security app such as McAfee, etc.

The table below mentions possible error codes that may be seen in the Print License Service log file

and a brief description of each.

Error Code

Description

Success.

-100

Something is wrong with the service. A security token

mismatch between the client and the server or internal

exceptions un- handled by the HPPLS.

-101

Null job handler.

-102

An invalid parameter is passed to any of the APIs.

-104

This corresponds to any exception that is thrown by HPPLS.

The details of the exception are logged in the log file.

-105

A valid session is already present, and client tries to

create another session.

-106

Either there are no features checked out and client tries to

query for feature details or there is no feature available with

the given name and version.

-107

An API is called with an empty session.

-108

Flexera server is not available.

-109

Flexera server is not responding.

-110

Some features that are passed to the API are invalid.

-111

Invalid file content.

-112

Token mismatch.

-113

Client does not exist.

-130

Flexera service is either not up & running or not ready to serve.

If Security Manager had been installed and the IP Address changed, that might cause issues as the

HP Print License service would be trying to contact the Flexera service as seen in the HPPLS.log file:

net.tcp//Old IP Address:8888/LicensingService

An uninstall and reinstall of HPSM would solve this issue.

A purchased license file must be ordered with the exact match of the Security Manager server MAC

address. The HP Print License manager will fail to operate properly without this exact match. If using

VMWare, make sure the appropriate virtual adapter MAC address is used. The physical mac address

of the NIC for the machine as seen under IPCONFIG is not the mac address of the VM.

Check the documentation of the VM vendor for instructions on how to find the mac address.

VMWare also recommends a static MAC assignment to accommodate software licensing scenarios.

If set to dynamic, it will not match the mac address in the license file any longer. Check the

documentation of the VM vendor for instructions on how to set to static.

If the mac address in the license is not what Security Manager expects, there should be an error in

the Flexera.log file found under C:\ProgramData\HP\HP Print License Service indicating what it read

in the license file and what it expects for the correct mac address:

10:15:13 (HPQ) Wrong hostid on SERVER line for license file: 10:15:13 (HPQ) C:\ProgramData\HP\HP

Print License Service\LicenseFiles\24092015101513_IPSC_DEVICES.lic

10:15:13 (HPQ) SERVER line says d8d38582501d, hostid is d8d38582501c 10:15:13 (HPQ) Invalid hostid

on SERVER line

10:15:13 (HPQ) Disabling 500 licenses from feature IPSC_DEVICES

If a trial/evaluation license installs fine but a purchased license does not install, it is likely that either

the mac address is wrong in the license file or the license file itself is bad. A purchased license

differs from a trial/evaluation license in that the purchased license only works on the machine

where the mac address is provided. That mac address must match the mac address in use, and if

running on a VM, the physical mac address for the machine is not the same as the mac address of

the VM.

Second, it is possible that the license file itself is bad or corrupt, although rare. The feature name in

the license file must be either IPSC_DEVICES or IPSC_DEMO. If it is HPSM_DEVICES, contact the

licensing support team and request another license. When doing so, make sure to instruct them to

generate a new license instead of re-sending the original license as they will not know the original

license is bad unless you tell them.

The license has readable text in the first half and encrypted content (SIGN=xxxx ...) in the latter half

that matches the readable content in the first half. Content of license files SHOULD NOT be

changed/edited at any time. Any edit to the readable content will no longer match the encrypted

portion and the file will become corrupted and unusable. If “SIGN=0” in the latter half of the file, it is

corrupt, and a new license file needs to be generated.

Make sure license files are present under:

C:\ProgramData\HP\HP Print License Service\LicenseFiles.

NOTE: By default, C:\ProgramData is a hidden folder. Either un-hide hidden folders to view this folder

or type the folder name (C:\ProgramData) directly in the address bar.

Device Status Errors and Credential Failures

It is important to understand how Security Manager interacts with devices to be able to

troubleshoot issues such as credential failures or device status errors.

Security Manager uses a variety of techniques to manage different types of devices depending

upon how features are exposed on the device.

• SNMP - Simple Network Management Protocol (SNMP) is used extensively in Security

Manager for all types of devices, especially during a Verify task to determine status. SNMP is

used to read from a device (SNMP GET REQ) or to write to a device (SNMP SET REQ). The

packet contains one or more Object Identifiers (OID) that defines the item being read or

configured. Security Manager supports both SNMPv1/SNMPv2, which is unencrypted, and

SNMPv3, which is encrypted.

• Web Services - Web Services (WS*) is a SOAP-based protocol used mostly with HP

FutureSmart devices. This communication uses port 7627. Security Manager communicates

over HTTPS to ensure that all data is encrypted during the transmission.

• LEDM - For the configuration of some non-HP FutureSmart devices, Security Manager uses

Low- end Data Model (LEDM). LEDM is based on the Representational State Transfer (REST)

style architecture, which is a design that describes a simple interface for transmitting XML

data over HTTP or HTTPS without an additional messaging layer. This configuration is done

over HTTP or HTTPS depending on the device configuration and device firmware.

• HTTP (“screen scraping” or “web scraping) – Some features are not exposed through typical

management protocols thus Security Manager resorts to performing what is referred to as

“screen scraping” to use HTTP to read a page and extract settings.

• DSMP - DSMP is a proprietary protocol that Security Manager uses for some configuration

options in the Digital Sending category for legacy HP Enterprise printers. DSMP is sent over

HTTP.

• PJL - Security Manager may use Printer Job Language (PJL) to test the PJL Password on

some devices.

• CDM - Security Manager uses a proprietary implementation of CDM only in version 3.1 and

beyond to accommodate the “Secure by Default” initiative released in HP FutureSmart 4.5

firmware. CDM is sent over HTTPS.

The Assessment Status and is defined as follows

Not assessed

Assessed and all settings in compliance

Assessed with only low risk items out of compliance

Assessed with medium risk items (and possibly low) items out of compliance

Assessed with high-risk items (and possibly low/medium) items out of compliance

It has nothing to do with the state or status of the device, it merely indicates which settings were

in or out of compliance during the last assessment task.

The Device Status column includes a visual icon to indicate Good (green check mark) or some

sort of error (red x) indicating a problem. It also includes a textual description of the error as

follows:

• Network Connection Error - this error indicates an issue trying to communicate with a device

over a specific protocol. Many times Security Manager is trying to securely connect to the

device over SSL/TLS and cannot for some reason. Most common reasons for this error

include the following:

o No response from device on basic network communications such as ping. Device may be

powered off or disconnected from network. Pings may be filtered at router or firewall.

o Device responds to pings but does not respond to Web Services (WS*) queries.

o Cannot browse to EWS page, perhaps EWS has been disabled.SSL/TLS handshake fails so

transaction cannot be encrypted. Operating system controls the TLS versions in the

handshake, not HPSM. Select the device and then select Do Not Enforce SSL/TLS.

o Certificate has MD5 hash which Microsoft no longer supports, so SSL/TLS handshake is

rejected. Right-click the device, select Set SSL\TLS Enforcement, and then select Do Not

Enforce to temporarily fix. Now Verify again to see if state clears. If so, for a permanent

solution, regenerate the self-signed certificate under EWS (might require newer Jetdirect

firmware) to generate a new certificate with a supported hash.

o ACL (Access Control List) blocking.

o No certificate support, already set to not enforce SSL/TLS (grayed out).

o Ports blocked, perhaps by firewall

o Some cases have been reported whereby DAT indicates SNMPv1/v2 passes for Gets and

Sets yet SNMPv3 cannot be enabled using SNMPv1/v2 and this error is generated. Often a

second remediation will clear the error.

• Connection Refused / Invalid Identity Certificate - If Security Manager installed an identity

certificate on the device, it tags it in the database to enforce trust for future communications.

o Removed, expired, or revoked certificates.

o Cannot connect to CRL (certificate revocation list) to check revocation.

• Credentials Failed - a mismatch occurred between what is stored in the database for the

specific credential and what is on the device. Security Manager will always try what is stored

in database first, then public/blank in case the credential was erased on the device, and finally

any credential stored in the Global Credential Store. If all three attempts fail, Security

Manager posts Credentials Failed as it needs proper credentials to communicate with the

device.

o SNMP behavior is to not respond to SNMP REQ packet when community name is wrong.

o Older devices had two locations under EWS for Admin Password.

• Device Not Supported - device does not support enough security related items to be deemed

supported by Security Manager. See “Supported Device List” whitepaper for a complete list of

supported devices.

• Error - this is a rarely seen state that seems to occur on the devices that answer so little to

queries that Security Manager deems it not manageable.

• Hostname Resolution Error - the hostname cannot be resolved. The DNS name server does

not recognize the hostname being presented by Security Manager that previously

represented the device.

• License Required for Assessment - not enough licenses are available, a license has not been

assigned to the device yet.

• No Information - a Verify task has not yet been performed where Security Manager gathers a

dozen or so attributes about the device and populates the columns. Adding a device will only

perform a hostname lookup, but the device will remain in No Information status until a Verify is

performed or an Assessment which begins with a Verify task.

Remember, browsing to EWS is strictly HTTP traffic. Security Manager and tools such as Web

Jetadmin use WS*, LEDM, SNMP, DSMP, etc. to communicate to devices for the various settings

depending upon how they are exposed for fleet management per device family.

For example, SNMP could be disabled on devices and not affect EWS browsing, but tools such as

Web Jetadmin and Security Manager would be severely hampered as they would not be able to

communicate with the device using a critical protocol.

Credential Management

It is common for both Web Jetadmin and Security Manager to be managing a fleet, and when one

tool changes device credentials, the other tool indicates a credential failure until it too knows the

credentials.

The best way to troubleshooting credential failures is to hone in on a single device, highlight one

device and check the Properties by clicking on the device IP Address link. Normally for devices with

no credential failure, the credentials will all be in a Valid state as seen to the right.

However, when the status indicates Credentials Failed, Security Manager does not know that

credential to read and/or write settings that require that credential.

SNMP

SNMP v1/v2 credentials are broken into two types: Read and Read/Write.

These equate to the Get Community Name and Set Community Name under EWS. If there are no

credentials entered for the Get Community Name in the EWS, and if the Disable SNMP v1/v2 default

Get Community Name of “public” checkbox is not selected, the device is wide open to read

information using SNMP with “public” used as the Community Name for an SNMP GET REQ packet.

This is not uncommon as very little if any sensitive data is passed thru Read attempts.

Starting with version 3.1, Security Manager checks both Set Community Name and a Get

Community Name during a Verify task, and if fails, it will post a credential failure. If either an SNMP

GET REQ or SNMP SET REQ packet receives no response, it is assumed a credential failure is

present because devices will not respond if a Community Name does not match.

To resolve this situation, either clear the credentials on the device under EWS, or add the SNMP

v1/v2 Read/write Community Name or the SNMP Write Community Name to the database

(credential store) for the device by selecting it, clicking the Set Credentials icon, and then

selecting Configure.

Admin (EWS) Password

If the Admin (EWS) Password is claiming Credentials Failed, this means that the test Security

Manger performs to determine if an Admin (EWS) Password is present is failing. For HP FutureSmart

devices, Security Manager will attempt to use web services to retrieve a system configuration page.

Proper EWS credentials are required to retrieve such a page.

If the page is not returned, it is assumed the Admin (EWS) Password that Security Manager has

stored for the device in the database or in the global store is incorrect. For older non-FutureSmart

devices, an attempt is made over an HTTP request to retrieve the Security Status page under EWS.

Again, if the page is not returned, it is assumed the Admin (EWS) Password that Security Manager

has stored for the device in the database or in the global store is incorrect.

One technique to resolve this issue is to add the Admin (EWS) Password to the database (credential

store) for the device by selecting it, clicking the Set Credentials icon, and then selecting Configure.

Another technique is to clear the Admin (EWS) Password under EWS, then right-click the device in

HPSM and clear the credentials that are stored in the database under Set Credentials, Reset.

This ensures that Security Manager and the EWS match. Perform a Verify task and see if

credentials failures are cleared. If it still claims credentials failed, try deleting the device and

rediscovering.

Example of Check Credentials

Here is an example of a typical device interrogation of an HP FutureSmart device to check

credentials.

During the Verify task, Security Manager starts by pinging the device. If there is no response to

pings, a Network Communication Error status is posted.

• If pings are successful, there are several SNMP GET REQ packets sent with various OIDs

to retrieve basic device information.

• If the OIDs receive no response, Security Manager makes an educated guess that the

SNMP credentials are wrong since that would be the exact behavior if the Community

Names do not match, and the device still responds to pings.

Therefore, a Credentials Failed status is posted blaming the SNMP Read Community Name. The

remaining transactions are secure ones over port 443 and port 7627 using to attempt to retrieve

an endpoint called SystemConfiguration over web services.

If that fails, there will be several indications in log files that it failed, and the status will indicate

Credentials Failed as seen below blaming the Admin (EWS) Password because you must know the

correct Admin (EWS) Password to retrieve the information via Web services:

Figure: Device Properties, Credentials tab

The EapNetworkLib.log file contains statements indicating the request was forbidden, for example:

15:32:47,213 ERROR EapNetworkLib [28] -

EapNetworkLib.EapUnauthorizedAccessException: XmlRest.Get(NPI851843,

/systemconfiguration, 0) failed. HTTP status code: Forbidden ---> System.Net.WebException: The

remote server returned an error: (403) Forbidden.

If you try to access the device EWS, it should only show the Information tab unless you login and

provide the correct Admin (EWS) Password if an Admin (EWS) Password is set.

If one is set and you can successfully login to EWS by entering it, that same Admin (EWS) Password

can be manually entered into the Security Manager database for the device by right clicking the

device and then adding it:

Figure: Typing the Admin EWS Credentials

Now a verify task should succeed because the password matches the device.

Device Communication Log Files

If failures still cannot be resolved for SNMP Community Names or Admin (EWS) Password, view the

log files for more information. Log files are stored under C:\Program Files (x86)\HP Security

Manager\logs.

The three log files containing the most valuable troubleshooting data include:

• HPSM_Service.log – contains data regarding HPSM service and actions it performs

• EapNetworkLib.log – contains data regarding network traffic to devices

• EapDeviceLib.log – contains data specific to device settings

The EapNetworkLib.log file contains statements such as below that either the SystemConfiguration

endpoint cannot be retrieved, or the Security Status page cannot be returned for Admin (EWS)

Password issues. It also contains statements that SNMP GET REQ or SNMP SET REQ attempts were

not returned for SNMP credential issues.

Example Admin (EWS) password failure for HP FutureSmart device:

Exception message=XmlRest.Get(15.86.190.69, /systemconfiguration, 0) failed. HTTP status code:

Forbidden, inner exception message=The remote server returned an error: (403) Forbidden.

Example Admin (EWS) password failure for non-FutureSmart device:

15:52:04,451 WARN EapNetworkLib [4] - Exception

message=Web.Get(15.86.190.170,https://15.86.190.170:443/hp/jetdirect/s ecurity_status.html,0)

failed. HTTP status code: Unauthorized, inner exception message. The remote server returned an

error: (401) Unauthorized.

Example SNMP Set Community Name failure for non-FutureSmart device:

Exception message=Snmp.Set(15.86.190.69,1 varbind(s)) timed out., inner exception message=No

response was received from the agent. Timeout is set to 30000 milliseconds.

Credentials not Validated or Incorrect

One status that might display is “Credentials not Validated or Incorrect”.

When clicking on the IP address and selecting the Credentials section, HPSM will display a status of

Couldn’t Verify for the Admin credentials.

In the EAPNetworkLib.log the following error might be present:

2021-10-14 11:00:36,938 ERROR EapNetworkLib [94] - uid=_dd999c08fa2f_192.168.178.75, Aggregate

exception=[type=IOException:message=Authentication failed because the remote party has

closed the transport stream.]

2021-10-14 11:00:36,938 ERROR EapNetworkLib [94] - uid=_dd999c08fa2f_192.168.178.75, Exception

message=XmlRest.Get(192.168.178.75, /systemconfiguration, 0) Caught a WebException

communicating with address=192.168.178.75 Exception of type 'EapNetworkLib.EapWebException'

was thrown. , inner exception message=Exception of type 'EapNetworkLib.EapWebException' was

thrown.

In the EapDeviceLib.log the following error might also be present when it was an instant-on

announcement:

2021-10-14 15:44:13,936 ERROR Pipeline [4] - uid=_ef385274a315_15.23.155.27, Pipeline.Execute(): Network

connection refused, address=15.23.155.27

When trying to open the EWS page with Internet Explorer (IE) directly from the HPSM server you might

receive the error ‘This page can’t be displayed’:

This error message indicates a mismatch of TLS and ciphers between the server/IE and the device.

When the EWS page of the device can be displayed successfully in IE from the HPSM server, then

HPSM should be able to return a device status of Good.

When trying to open the EWS page with Chrome from the HPSM server, the printer might get

displayed correctly.

HPSM uses the same cypher settings as IE and the OS. Chrome seems to bypass some of those

settings.

On HP FutureSmart devices the active ciphers and TLS settings are displayed on the Networking tab

under the Secure Communication menu.

In IE, open Internet Options, and select the Advanced tab to view or edit the SSL and TLS selections:

However, If the ciphers are not enabled in the OS, then they cannot be enabled via IE. The OS

SChannel and cipher suites are controlled by the Schannel registry settings and possible group

policy settings:

HKLM SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL

With IIS Crypto, see https://www.nartac.com/Products/IISCrypto/Download, you can easily visualize

the settings.

For successful EWS password validation there must be a match between the OS settings and device

settings for the SSL/TLS Protocols and the Active Ciphers on the device.

NOTE: OS changes for the SSL/TLS protocol and Ciphers require a server reboot.

It is also possible to check in a browser which Protocol and Cipher is used when communication is

successful. For Chrome, click the 3 dots in the upper right corner, select More tools, Developer Tools,

Security.

Hanging or Slow Tasks

If a task is started but never seems to complete or takes an extraordinary time to complete, there

may very well be a valid reason or some configuration options to try to improve performance.

Hung Tasks vs. Slow Tasks

It is important to determine if the task is hung forever and will never complete or if it is taking

much longer than expected to complete.

First, try the same policy on a few devices to see if it completes. Also, try an extremely simple task

on the fleet to see if a setting in the policy might be responsible for the delay or hang. You want to

narrow down settings and/or devices to see if they are causing a task to be slow or hung.

Versions of Security Manager prior to 3.0.1 had known issues where certain items in a policy

could cause a task to hang indefinitely.

For example, setting that required web scraping to assess/remediate relied upon a Microsoft

library to perform the HTTP transaction. Some servers had Microsoft libraries installed that

would not permit performing this HTTP transaction. Security Manager 3.0.1 starting using a third-

party library to perform these transactions that eliminated the hang.

Security Manager relies on a shared library called Microsoft.MSHTML.dll to perform queries on

devices using a technique referred to as “web scraping” to manage the features. Older devices

rely on this technique more than newer HP FutureSmart devices. Typically, this file is already

present in the Global Assembly Cache (GAC), and if so, Security Manager will use the file loaded in

GAC. If the file is not present in GAC, Security Manager will load a copy of this file.

Issues may arise where functionality of this file is not working or is blocked by browser settings,

for example. One such setting is a browser setting called “Run antimalware software on ActiveX

controls” under the Security tab that will block the usage of this dll. If so, there is a possibility that

a task relying on this .dll file to perform managing of a feature may hang as there is never a return

to the query. Security Manager 3.1 uses a newer third-party library to perform web scraping to

alleviate this issue on these unique servers that block the usage of the MS library.

If tasks are still hanging, deleting the task under the Tasks tab will not stop all the devices from

being assessed/remediated. The devices are still tagged in the database as not complete and will

start again if the service is restarted, for example. You will have to wait for incomplete devices to

run their course.

Credential Failed or Network Connection Error Impacting

Performance

Devices in a Credentials Failed or Network Connection Error status can cause extreme delays in

completing tasks. Try to run the task again without devices in this state. If it completes in a more

reasonable amount of time, Security Manager 3.1 offers some timeout and threading settings that

can be configured to help lessen to negative performance impact of devices in such a state.

These settings can be controlled using configuration items in the HPSM_Service.exe.config file. If

the server has the power, increasing the threads and decreasing the timeouts can substantially

reduce the time it takes to complete tasks. In the list below several change suggestions are listed.

If performance or behavior is worse after making those changes, then you need to revert the

changes.

To change these parameters, open the following file in a test editor:

C:\Program Files (x86)\HP Security Manager\HPSM_config.exe.config

Edit the following entries, save changes, and restart the Security Manager service.

<add key="snmpRequestTimeout" value="30000"

Value is in milliseconds (30s) – amount of time to wait for responses to SNMP packets

Change suggestion: In most environments you can set this to 2000

<add key="verificationSnmpRequestTimeout" value="2000"

Value is in milliseconds (2s) – amount of time to wait for SNMP responses during a Verify task

<add key="timeBetweenEapRetry" value="5000"

Value is in milliseconds (5s) – time to wait between retries on SNMP packets

Change suggestion: in most environments you can set this to 2000

<add key="eapRetryLimit" value="2"

Number of retries if a device fails to respond for a request

Change suggestion: Limit the retries to 1

This setting is suggested when there are devices ending up in Inconclusive or Admin credentials

Failed

<add key="eapMaxThreadCount" value="100”

Total number of active threads to devices

HPSM 3.6.1 and older was ignoring the configured value.

HPSM 3.7 and newer will be using the actual configured value. It is therefore recommended to start

with 100. Check performance and after that increase and validate if this improves the performance.

<add key="maxNumberTasks" value="10"

Number of tasks that can be open simultaneously. This means that Security Manager will either

have 10 child tasks of 25 devices each open at a time for Verification and Assessment, or 10 tasks of

one device each During Instant-On, Remediation and Credential retry.

Change suggestion: increase this to 50.

<add key="numberDevicesInEAPTaskCheckpointInterval" value="25"

This only applies to verification and assessment/remediation tasks, it means 25 devices will be

present in each child task.

Change suggestion: increase this to 50.

<add key= “caManagerMaxThreadCount" value="100"

Number of devices at a time when a request is sent to CA manager to provide certificate

remediations.

Change suggestion: increase this to 200.

Slow device remediation, EWS Password not verified, ping and SNMP

are working fine

Sometimes there are some issues accessing the EWS page of the device or accessing the device

over port 7637 while ping and SNMP are working fine. This can cause major delays while trying to

remediate those devices.

For every config item which uses web services, the default time out of 60 seconds will be used. From

HPSM 3.6 onwards this timeout is configurable in EapNetworkLib.dll.config.

Recommendation is to change this timeout to 20 seconds and monitor if this brings indeed faster

assessment and remediation.

1. Open the EapNetworkLib.dll.config file

2. Change the value for the httpTimeout to for example 20 seconds

3. Restart the HPSM service

4. Monitor the performance and make further adjustments if needed.

Performance Impact of Instant-On Remediations

A high volume of Instant-On automatic remediations occurring in the background will absolutely

affect performance. It is possible there are a few devices causing such a high volume, or it might be

devices are sending announcements for legitimate reasons and the server can only handle so many

tasks.

If many of the messages are coming from one device, it might be because a faulty device is

operating from the network and coming back again, or that devices are constantly rebooting for

some reason. Try eliminating such devices if suspicions arise, they may be responsible for the bulk

of Instant-On remediations.

You can also try turning off Instant-On completely in Security Manager, at least temporarily to

see if it is the contributing factor of the hangs/delays.

View the number of active Instant-On assessments under the Tasks tab by selecting Instant-On

Tasks:

Instant-On announcements are processed immediately (i.e adding of the devices), but the action

of performing the assessment/remediation task is scheduled.

These display as one task at a time for each device, and Security Manager processes a

maximum of 10 Instant-On remediation tasks at a time.

This number of maximum tasks at a time can be controlled using a configuration item in the

HPSM_Service.exe.config file found under

\Program Files (x86)\HP Security Manager

Change “maxNumberTasks” from a value of “10” to as much higher value to see if it makes a

difference.

The threading for scheduled tasks can also be increased here. By default, each parent task is

broken into child tasks of 25 devices each. This number can be increased to increase the

performance by changing ="numberDevicesInEAPTaskCheckpointInterval" to a higher

number.

It may boil down to if the fleet is exceptionally large and Instant-On seems to be consuming

the bulk of the bandwidth, a separate server may be required to process the Instant-On tasks.

This will allow for the “instant” remediation benefit of Instant-On in cases where a device is

cold reset, for example.

However, it will not compromise the ability of the scheduled tasks to keep the entire fleet in

compliance if running on a separate server.

One other possible cause of hanging tasks includes the Email Summary Reports feature.

Assess/remediate tasks run forever if an invalid email address is configured in automated

output. In such a case, tasks should be canceled manually, and the correct email address

should be configured in settings.

Suggestions for improving performance are provided below but certainly not limited to these

values. This may take some trial and error. For example, setting eapRetryLImit=0 can shave

approximately 24s off total assess/remediation time per device.

Slow performance and reaching max SQL connection pool with

Timeouts in HPSM_service.log

Issue: when HPSM needs to handle lots of instant on requests and scheduled tasks, it will need a

lot of SQL connections. By default HPSM is using an SQL connection pool for 100 simultaneous

connections and a SQL timeout of 60 seconds. When this is not enough timeouts will occur, as

you can see in the following entries from the HPSM_service.log file.

2023-06-07 07:25:08,410 WARN NHibernate.Util.ADOExceptionReporter [7855] -

System.InvalidOperationException: Timeout expired. The timeout period

elapsed prior to obtaining a connection from the pool. This may have

occurred because all pooled connections were in use and max pool size

was reached.__ at

System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnect

ion owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions

userOptions, DbConnectionInternal oldConnection, DbConnectionInternal&

connection)__ at

System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal

(DbConnection outerConnection, DbConnectionFactory connectionFactory,

TaskCompletionSource`1 retry, DbConnectionOptions userOptions)__ at

System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1

retry)__ at

System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1

retry)__ at System.Data.SqlClient.SqlConnection.Open()__ at

NHibernate.Connection.DriverConnectionProvider.GetConnection()__ at

NHibernate.AdoNet.ConnectionManager.GetConnection()__ at

NHibernate.AdoNet.AbstractBatcher.Prepare(IDbCommand cmd)

2023-06-07 07:25:09,684 ERROR NHibernate.Util.ADOExceptionReporter

[5367] - Timeout expired. The timeout period elapsed prior to

obtaining a connection from the pool. This may have occurred because

all pooled connections were in use and max pool size was reached.

Solution: increase the SQL connection pool for Nhibernate and SQL database connection in the

HPSM_service.exe.config file and the Web.config file and restart. Steps:

1. Open the HPSM_service.exe.config file and the Web.config file.

2. Search for the following entries and add Max Pool Size=200 at the end of the connection

string

<add key="dbConnection" value="Server=(local)\SQLEXPRESS2022;initial

catalog=HPIPSC;Integrated Security=SSPI;Connection Timeout=30;Max Pool

Size=200" />

<add key="dbMasterConnection"

value="Server=(local)\SQLEXPRESS2022;initial catalog=master;Integrated

Security=SSPI;Connection Timeout=30;Max Pool Size=200" />

<property name=

"connection.connection_string">Server=(local)\SQLEXPRESS2022; initial

catalog=HPIPSC;Integrated Security=SSPI;Max Pool Size=200</property>

3. Save the changes

4. Stop HPSM service, stop the HPSM application pool

5. Start HPSM application pool

6. Start HPSM service.

Note: if the SQL timeout is not listed in the keys dbConnection and dbMasterConnection, then it

will be using the default timeout of 30 seconds. Besides those sql timeouts there are also a few

other SQL timeouts defined in the config files:

<!--SqlQueryTimeout: Sql query timeout in seconds. 0 indicates no

limit (an attempt to execute a command will wait indefinitely), if any

invalid value is given,Recommended value is 300 will be taken-->

<!--MaintenanceTaskQueryTimeout: Maintenance task query timeout in

seconds. 0 indicates no limit (an attempt to execute a command will

wait indefinitely), if any invalid value is given, Recommended value is

600 will be taken-->

<session-factory name="NHibernate.Test">

Note: it’s possible to monitor the actual sql connection pool with Performance Monitor:

Start Performance Monitor

Click on Performance Monitor under Monitoring Tools in the left pane

Right-mouse click on the graph in the right pane and select Add Counters

Double click on .NET CLR Data

Add the desired SqlClient counters, see screenshot.

It’s also possible to run a query on the SQL server to see the current active number of

connections. Open SQL Server Management Studio, connect to the instance and select New

Query. Add the following query:

SELECT COUNT(*) AS ConnectionCount

FROM sys.sysprocesses sp

JOIN sys.databases db ON sp.dbid = db.database_id

WHERE db.name = 'HPIPSC'

Hit Execute

Note: this only displays the current number of connections and can be used to check the number

of connections when HPSM UI is slow.

Upgrade Issues

‘Invalid Task: No device specified’ after upgrading to HPSM 3.7

After upgrade to HPSM 3.7 tasks might not run and end-up with error Invalid Task; no device

specified.

Figure: Errors listed on the Tasks tab of HPSM

The HPSM_service.log file is containing errors, such as Cannot find data type UDT_DeviceId and

Could not find stored procedure ‘HPIPSC.DBO.spDeleteSingleTask.

See following examples:

2022-02-16 11:59:22,611 DEBUG Service [5] -

CheckAndMaybeThrowDBConnectionOrFullException: System.Data.SqlClient.SqlException

(0x80131904): Column, parameter, or variable @T_DeviceIds. : Cannot find data type

UDT_DeviceId.__ at System.Data.SqlClient.SqlConnection.OnError(SqlException exception,

Boolean breakConnection, Action`1 wrapCloseInAction)__ at

System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject

stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)__ at

System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand

cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler,

TdsParserStateObject stateObj, Boolean& dataReady)__ at

System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()__ at

System.Data.SqlClient.SqlDataReader.get_MetaData()__ at

System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior

runBehavior, String resetOptionsString, Boolean isInternal, Boolean

forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)__ at

System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior,

RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task,

Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean

describeParameterEncryptionRequest)__ at

System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior,

RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1

completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean

inRetry)__ at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior

cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)__ at

System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String

method)__ at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[]

datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command,

CommandBehavior behavior)__ at System.Data.Common.DbDataAdapter.Fill(DataSet

dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command,

CommandBehavior behavior)__ at System.Data.Common.DbDataAdapter.Fill(DataSet

dataSet, String srcTable)__ at

LocksmithBusinessLogic.BizLogicMgrHelper.GetDeviceDataSet(QueryFilter q, Boolean

includeAllDevices, Boolean onlyReturnRecCount, LSGroupVO

group)__ClientConnectionId:8b38d659-bb01-481c-99e1-ce44e49af760__Error

Number:351,State:3,Class:16

This means that the stored procedures and User-Defined Table Types were not created correctly

during the upgrade to 3.7.

The following figure shows how it should look after a successful upgrade to version 3.7:

Figure: User-Defined Table Types created correctly and listed under Types

If the user who has been running the SQL upgrade commands did have a default database other

than DBO, then this issue would occur.

Solution

Upgrade to HPSM 3.7.1.

Obtain the updated InstallSQLScripts_3.7VersionC.zip and re-run the

InstallOrUpgradeRemoteDb.bat file to create the missing stored procedures and the missing

User-defined table types.

Workaround

Change the default schema for the user running the SQL upgrade to DBO and re-run the

InstallSQLScripts.

Figure: ‘dbo’ is listed as the default schema in Login Properties under User Mapping

Web page not displayed, hangs, or Invalid Column name error after

upgrading to HPSM 3.7

In some situations, the HPSM UI will not display, the web page will not open, and/or the browser

hangs with a spinning icon while waiting to load the UI.

An ‘Invalid Column name’ error is listed in the HPSM_service.log file after upgrading to 3.7:

2 022-02-09 15:44:20,899 ERROR NHibernate.AdoNet.AbstractBatcher

System.Data.SqlClient.SqlException (0x80131904): Invalid column name

'RemoveLicenseAlone'.__ at System.Data.SqlClient.SqlConnection

The new column "RemoveLicenseAlone" in the Server.config table has not been created.

If you look at the UpgradeDbSchemaVer15to16.sql, you will find the following steps for this:

'============================================================================='

PRINT 'Add and Update RemoveLicenseAlone in ServerConfigTable'

'============================================================================='

IF NOT EXISTS(SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME =

'ServerConfigTable' AND COLUMN_NAME = 'RemoveLicenseAlone')

BEGIN

DECLARE @SQL NVARCHAR(MAX) = N'';

SET @SQL += N'

ALTER TABLE ServerConfigTable ADD RemoveLicenseAlone BIT NOT NULL DEFAULT 0'

PRINT @SQL

EXECUTE(@SQL)

END;

The execution of the part failed during the upgrade process. Therefore, it is required to run the

following sql commands to create the missing table (this can be done from SQL Management Studio):

IF NOT EXISTS(SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME =

'ServerConfigTable' AND COLUMN_NAME = 'RemoveLicenseAlone')

BEGIN

DECLARE @SQL NVARCHAR(MAX) = N'';

SET @SQL += N'

ALTER TABLE ServerConfigTable ADD RemoveLicenseAlone BIT NOT NULL DEFAULT 0'

After making the change to the table, restart the HPSM service.

T-SQL Execution command line utility has stopped working error

While running the InstallOrUpgraderemoteDB.bat you might experience an SQLCMD crash,

where the following error is displayed:

T-SQL Execution command line utility stopped working

Figure: “T-SQL Execution command line utility stopped working” error during InstallSqlScripts

Solution

After clicking Close the program the InstallOrUpgradeRemoteDb.bat will continue to run, but

one or more stored procedures might be missing.

1. Make sure the following Stored Procedures and User-defined table types have been installed

after upgrade:

2. If a stored procedure is missing, double click the sql command in the InstallScripts.zip file, for

example: CreateStoredProcedure_DeleteAllDeletedTasksAndReferencedRecords.sql

This will open the sql command in SQL Server Management Studio.

NOTE: You can also open the sql command in Notepad and copy and paste the command in

SQL Server Management Studio.

3. Change the USE $(DBNAME) into the actual database name (USE HPIPSC, for example).

4. Right-click and select Execute.

The missing stored procedure should be created.

If it is still not created, you must continue with the steps provided in the solution described in the

section “HPSM service stopping automatically after upgrading to HPSM 3.7 or HPSM 3.7.1 with

ERROR Invalid column in HPSM_service.log”.

Application Error in Event log while executing the InstallSqlScripts

In the event log (under Windows Logs, Application) the following error is displayed while

executing the InstallSqlScripts:

Application Error with event 1000

Solution

Follow the same solution steps provided for

T-SQL Execution command line utility has stopped

working error

HPSM service stops, http Error 503 in browser, and Invalid column

error after upgrading

When the HPSM service is stopping automatically and an http Error 503 is returned in the browser

after upgrading from HPSM 3.5 to a newer HPSM version (3.6, 3.6.1, 3.7, or 3.7.1), an “Invalid column

name error” displays in the HPSM_service.log file:

2022-03-08 21:18:02,058 ERROR NHibernate.AdoNet.AbstractBatcher

System.Data.SqlClient.SqlException (0x80131904): Invalid column name

'RemoveLicenseAlone'.__ at System.Data.SqlClient.SqlConnection.OnError(SqlException

exception, Boolean breakConnection, Action`1 wrapCloseInAction)__ at

System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject

stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)__ at

System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand

cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler,

TdsParserStateObject stateObj, Boolean& dataReady)__ at

System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()__ at

System.Data.SqlClient.SqlDataReader.get_MetaData()__ at

System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior

runBehavior, String resetOptionsString, Boolean isInternal, Boolean

forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)__ at

System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior,

RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task,

Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean

describeParameterEncryptionRequest)__ at

System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior,

RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1

completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean

inRetry)__ at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior

cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)__ at

System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String

method)__ at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand

cmd)_

ClientConnectionId:ca4ef007-8d35-4077-957c-5ed4ec981bce

_Error

Number:207,State:1,Class:16

This means that the column ‘RemoveLicenseAlone’ has not been created in the

ServerConfigTable during upgrade.

Solution when using HPSM 3.7.1

Run the installSQLscripts manually. You can run the HPSM installer to extract the

InstallSQLscripts or you can find them in the HPSM root installation directory, such as:

C:\Program Files (x86)\HP Security Manager\

Unzip the file and re-run the InstallOrUpgradeRemoteDB.bat file, see the readme inside the

InstallSQLscripts for further details.

NOTE: This only seems to happen when using SQL 2012 or older in combination with HPSM 3.5

installed on d:\ or e:\ drive and default schema not set to DBO during upgrade of HPSM.

Restart the HPSM application pool and start the HPSM service after making this change to the

table.

Solution when using HPSM 3.7

When using HPSM 3.7, you must obtain the InstallSQLScriptsC from HPSM support. To do this,

use one of the following methods:

NOTE: Both methods will create the missing column RemoveLicenseAlone in the ServerConfig

table.

Method One

1. Unzip the file.

2. Re-run the

InstallOrUpgradeRemoteDB.bat

file.

NOTE: For instructions, see the Readme inside the InstallSQLscripts.

3. If the issue persists, make the following additional changes to the scripts:

a. Unzip the InstallSQLScripts.zip (or InstallSQLScriptsC.zip when using 3.7.0):

b. Open the file CreateStoredProcedure_MaintenanceTasks.sql with Notepad.

c. Add the following commands to the second line of the file:

EXEC sp_configure 'show advanced options', '1'

RECONFIGURE

-- this enables xp_cmdshell

EXEC sp_configure 'xp_cmdshell', '1'

RECONFIGURE

Figure: Correct configuration in Notepad

4. Save the changes.

5. Re-run the

InstallOrUpgradeRemoteDB.bat

file again.

NOTE: The syntax for this batch file is described in the corresponding

Readme_InstallSqlScripts.zip.

6. Restart the HPSM application pool and start the HPSM service after making this change.

Method Two

1. Run the following sql command from SQL Management studio when using a database called

HPIPSC:

use HPIPSC

IF NOT EXISTS(SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME =

'ServerConfigTable' AND COLUMN_NAME = 'RemoveLicenseAlone')

BEGIN

DECLARE @SQL NVARCHAR(MAX) = N'';

SET @SQL += N'

ALTER TABLE ServerConfigTable ADD RemoveLicenseAlone BIT NOT NULL DEFAULT 0'

PRINT @SQL

EXECUTE(@SQL)

END;

2. Restart the HPSM application pool and start the HPSM service after making this change

to the table. The missing column RemoveLicenseAlone is listed in the ServerConfig table.

Figure: ServerConfig table with the RemoveLicenseAlone column listed

Task Error: Internal error in processing when re-running a task

after upgrading to HPSM 3.8 and/or unable to complete a new task.

Immediately after upgrading to HPSM 3.8, it’s possible that you cannot re-run a task. As it will end

up with Task Error. See screenshot:

When creating a new task, the newly created task will never complete and cannot be stopped

from the UI.

Instant on announcements cannot be handled by HPSM.

In the HPSM_service.log file you might see Nhibernate ERRORS:

2022-07-05 10:58:50,860 ERROR NHibernate.AdoNet.AbstractBatcher

System.Data.SqlClient.SqlException (0x80131904): Invalid Column name

‘LastSuccessfulAutoEWSPasswordResetOn’.__

2022-07-05 10:58:51,095 ERROR NHibernate.Util.ADOExceptionReporter [3] – Invalid column

name ‘LastSuccessfulAutoEWSPasswordResetOn’.

In the InstantOn.log you might see ERRORS like:

2022-07-05 12:19:50,689 ERROR InstantOn [4] - DirectSSLListenerTask:

LocksmithCore.ExceptionPackagerHdap: Unable to find or create a device__ bei

LocksmithBusinessLogic.PackagerNAP.ProcessDeviceDiscoveryReturnDevice(DeviceIdentity

deviceIdentity, Boolean& newDeviceWasCreated)__ bei

TaskManager.DirectSSLListenerTask.DoWork(Object data)

Both issues are caused by the fact fact that the column

LastSuccessfulAutoEWSPasswordResetOn in the dbo.deviceTable was not created during

upgrade.

Solution:

(re)run the InstallOrUpgradeRemoteDB.bat from the InstallSqlScripts.zip file to create the

missing column in the database.

Note: the InstallSqlScripts are located in the directory Program Files (x86)\HP Security Manager

Restart he HPSM service.

If the task still fails:

• Stop the HP Security Manager service

• Run the following SQL script from SQL Management Studio (change HPIPSC into a

different database in case it’s named differently:

USE HPIPSC

IF NOT EXISTS(SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME =

'DeviceTable' AND

(COLUMN_NAME = 'LastSuccessfulAutoEWSPasswordResetOn'))

BEGIN

DECLARE @SQL NVARCHAR(MAX) = N'';

SET @SQL += N'

ALTER TABLE dbo.DeviceTable

ADD LastSuccessfulAutoEWSPasswordResetOn datetime NOT NULL DEFAULT

CONVERT(DATETIME, ''9999-12-31 23:59:59.000'')'

PRINT @SQL

EXECUTE(@SQL)

END;

• Restart HPSM Security Manager service

Task Error: After upgrading to HPSM 3.10 some tasks end up in

Task Error

When running a task with a policy which has been upgraded to 3.10 and which contains an ID certificate,

you might see the following error in the HPSM_service.log file:

2023-07-24 03:31:40,024 ERROR Data

PolicyFramework.LocksmithPolicyException: : Duplicate resource id's

found in policy. Can't resolve reference: PLSanOptions

Line 4183: 2023-07-24 03:31:40,024 ERROR Service [6] - AssessmentTask:

Task Error:

{0}: Duplicate resource id's found in policy. Can't resolve reference:

PLSanOptions at

PolicyFramework.PolicyResourceManager.GetResource(String

resourceName)__ at

PolicyFramework.PolicyResourceManager.GetResource(ItemId itemId)__ at

PolicyFramework.Policy.IsActiveInstance(ItemId itemIdentity)__ at

PolicyFramework.Policy.GetControlValue(ItemId id,

PolicyControlValueType type)__ at

AssessmentRemediation.AssessmentParser.IsActive(ItemId identity,

List`1& returnedPolicyIds)__ at

AssessmentRemediation.AssessmentParser.GetActivePolicyItem(String

itemName, List`1& retPolItem)__ at

AssessmentRemediation.AssessmentParser.CreateAssessmentItems()__ at

AssessmentRemediation.Assessment..ctor(Policy policy)__ at

AssessmentRemediation.AssessmentTask.ExecuteTask()

2023-07-24 03:31:40,024 ERROR Service [6] - Warning - Truncated DB

string object - TaskBase - ErrorMessage -orginal: AssessmentTask: Task

Error: {0}

Solution: create a new policy with exactly the same values and run the task with the new policy.

Additional information: the policy was not upgraded correctly. A root cause was never found as

the original DB (before upgrade) was no longer available when this issue was reported.

Growing Database and Nightly Maintenance failing

Growing Database and Nightly Maintenance failing (3.6.1 and older)

The HPSM inbuild maintenance starts at 01.00 AM.

If the database keeps growing (for example over 10GB), then you need to verify if the nightly

maintenance is being executed correctly. To do this, follow these steps:

1. Check the HPSM_service.log file for any errors that indicate the nightly maintenance is

failing. For example, an SQL timeout might occur.

NOTE: Some information is only logged as INFO not as ERROR.

2. Configure the timeout value. From 3.5 onwards this can be configured with the

configuration parameter in the HPSM_Service.exe.config file:

NOTE: The default timeout for the maintenance task is 30 minutes.

3. After making changes, restart the HPSM service (required).

Because it is unknown which timeout is required, it is better to remove the old data with a

script from SQL Management Studio to prevent timeouts.

To do this, follow these steps:

1. Stop the HPSM service.

2. Execute the following script:

DECLARE @X INT=1;

WAY:

SELECT TOP 10000 * into #NEWTABLE FROM

(SELECT rec.ID AS recID, rToret.KEY_ID as rToretID, rt.ID AS rtID, rvt.ID as rvtID, rTorv.ID AS

rTorvID,

rTorat.KEY_ID AS rToratKEY_ID, rat.ID AS ratID, av.ID AS avID, raTop.ID AS raTopID

FROM DBO.RecommendationTable rec

LEFT OUTER JOIN DBO.RecToReasonsTable rToret ON rToret.KEY_ID = rec.ID

LEFT OUTER JOIN DBO.ReasonTable rt ON rt.ID = rToret.Reason

LEFT OUTER JOIN DBO.ReasonToReasonValuesTable rTorv ON rTorv.ID = rt.ID

LEFT OUTER JOIN DBO.ReasonValueTable rvt ON rvt.ID = rTorv.ReasonValue_ID

LEFT OUTER JOIN DBO.RecToRecommendationActionsTable rTorat ON rTorat.KEY_ID = rec.ID

LEFT OUTER JOIN DBO.RecommendationActionTable rat ON rat.ID =

rTorat.RecommendationAction

LEFT OUTER JOIN DBO.AssessmentValueTable av ON av.ID = rat.ActionValue_REF

LEFT OUTER JOIN DBO.RecActionsToParametersTable raTop ON raTop.ID = rat.ID

where rec.AssessmentAndPolicyUniqueID NOT IN ( select distinct

dal.assessmentAndPolicyUniqueID as uniqueID from DBO.DeviceAssessmentLogTable dal

where dal.State = 2 )) as Sub1

--select count (*) from #NEWTABLE

DELETE a FROM DBO.RecToRecommendationActionsTable a INNER JOIN #NEWTABLE B ON

a.KEY_ID= B.rToratKEY_ID

DELETE a FROM DBO.RecToReasonsTable a inner join #NEWTABLE B on a.KEY_ID = B.rToretID

DELETE a FROM DBO.RecommendationTable a inner join #NEWTABLE B on a.ID = B.recID

DELETE a FROM DBO.ReasonToReasonValuesTable a inner join #NEWTABLE B on a.ID =

B.rTorvID

DELETE a FROM DBO.ReasonTable a inner join #NEWTABLE B on a.ID = B.rtID

DELETE a FROM DBO.ReasonValueTable a inner join #NEWTABLE B on a.ID = B.rvtID

DELETE a FROM DBO.RecActionsToParametersTable a inner join #NEWTABLE B on a.ID =

B.raTopID

DELETE a FROM DBO.RecommendationActionTable a inner join #NEWTABLE B on a.ID =

B.ratID

DELETE a FROM DBO.AssessmentValueTable a inner join #NEWTABLE B on a.ID = B.avID

SET @X = (select count (*) from #NEWTABLE)

drop table #NEWTABLE

IF @X=10000 GOTO WAY;

3. If needed, shrink the database with the following command:

EXEC sp_configure DBCC Shrinkdatabase ('HPIPSC')

NOTE: After the script has been completed, there might be a lot of empty space in the

HPSM database. The space which the database needs will be a few GB smaller.

4. Restart the HPSM service.

From this point onwards the nightly maintenance should be finished within the default timeout of

30 minutes.

Growing Database and Nightly Maintenance failing (3.7 and newer)

After upgrade to 3.7, the nightly maintenance might fail and the Maintenance.log file contains

errors about missing procedures.

For example, the following error indicates that the stored procedure called

spRemoveOldRecommendations is missing from the SQL database.

2022-01-27 01:01:01,855 ERROR MaintenanceTask - RemoveOldRecommendation :

SqlException occured - Could not find stored procedure 'spRemoveOldRecommendations'

With SQL Management studio you can see all the stored procedures.

If one or more stored procedures are missing, then this might be caused by the original

upgrade/InstallSQLscripts of HPSM which uses a command which is only supported from SQL 2016

onwards.

The fixed scripts have been updated to support also older versions of SQL. The updated

readme_InstallSQLScripts contains the following version information:

Devices tab shows “Loading…” and is not showing any devices or

groups after upgrade

After upgrading it’s possible that the dashboard shows the correct number of devices, but on the

devices screen, the UI remains at Loading. See screenshot.

In the HPSM_service.log file you can find the following similar error:

2022-11-03 21:11:59,849 ERROR Service [38] -

ScheduledTaskMgr.Unregister - Unexpected Error:

System.InvalidCastException: Object cannot be cast from DBNull to

other types.__ at

System.DBNull.System.IConvertible.ToBoolean(IFormatProvider

provider)__ at

LocksmithBusinessLogic.BizLogicMgr.GetAllGroupRecordsAlone()__ at

LocksmithBusinessLogic.DashBoard.DashBoardManager.ExtractSnapShotDetai

lsFromDS(DataSet dsSnapShotList, DataSet notSupportedSnapShotList)__

at LocksmithBusinessLogic.DashBoard.DashBoardManager.SaveSnapShot()__

TaskManager.ScheduleTaskManager.CheckParentStateAndRemoveIfAppropriate

(Guid impactedParentID, Boolean forceDBCheck, String msg, Boolean

propagateStatusMsgAndStateToParent, TaskStatus finalTaskStatus)__ at

TaskManager.ScheduleTaskManager.UnregisterInstanceOnly(WorkerTaskState

100

wts_toUnReg, String msg, Boolean propagateStatusMsgToParent,

TaskStatus finalTaskStatus)__ at

TaskManager.ScheduleTaskManager.Unregister(WorkerTaskState

workerTaskState, String msg, Boolean propagateStatusMsgToParent,

TaskStatus finalTaskStatus)

This behavior is caused by the fact that there are multiple entries in the DB with All Devices

Groups with a value of NULL for the IsAutoGroup value. See screenshot.

There should only be one entry in this table for the All Devices Group with a value of zero for

IsAutoGroup.

To remove the incorrect entries from the table, please run the following script:

USE %DBNAME% -- Replace %DBNAME% with Database name

-- Delete records from GroupTable where IsAutoGroup is NULL

Delete from [dbo].[GroupTable] where IsAutoGroup is NULL

Note: This issue has originally been seen after upgrading from 3.6.1 to 3.8 on a database which

already contained the incorrect entries in 3.6.1. The root cause of the incorrect entries is

unknown. The only way to resolve this issue is by removing the incorrect entries.

After upgrading to HPSM 3.8 the Start Time of a newly created

task is always showing 12:00AM

When creating a new task on a server with localized settings, it’s possible that the Server Time

cannot be fetched by HPSM 3.8. The UI will in that case always display 12:00 AM as the starting

time of the new task.

When you run Chrome in developer mode (F12), you can also see the error message, like

(Central European Summer Time) Timepicker directive: "ng-model" value

must be a a Date object, a number of milliseconds since 01.01.1970 or

a string representing an RFC2822 or ISO 8601 date.

101

See screenshot:

Resolution: Upgrade to HPSM 3.9 or higher.

Workaound: Configure the HPSM server to use US settings (Region settings and language

settings)for Welcome Screen and new User Accounts.

After upgrading to HPSM 3.10 zero devices are displayed in

the UI and the event viewer shows and error with

CertExpiryDate in the description and/or Certificate Expiry

Date column missing

Issue 1: after upgrade to HPSM 3.10 and logging into HPSM there are zero devices displayed. In the

Windows event viewer, you can see a HP Security Manager error with event ID 0 and the following

description:

Error - Maybe a DB access issue - calling GetAllDevicesDataSet()

System.Data.SqlClient.SqlException (0x80131904): Invalid column name

'CertExpiryDate'.

102

Issue 2: after upgrade to HPSM 3.10 and logging into HPSM you cannot see the column option

Certificate Expiry Date when clicking on Column Organizer

Additional information:

When upgrading to HPSM 3.10 a new column called CertExpiryDate should get added to the

dbo.DeviceTable. If the column CertExpiryDate is missing, then zero devices will be displayed in

the UI (while in fact all the devices are still listed in the DB). When you re-run the InstallSQLscripts,

you can see the following error in the InstallOrUpgradeRemoteDB21.log:

ALTER TABLE dbo.DeviceTable

ADD CertExpiryDate datetime NOT NULL DEFAULT CONVERT(DATETIME,

'9999-12-31 23:59:59.000')

Msg 242, Level 16, State 3, Server gbrdsm050001522\BAR_SQLVIRT_DEV,

Line 2

The conversion of a varchar data type to a datetime data type resulted

in an out-of-range value.

Msg 1750, Level 16, State 1, Server gbrdsm050001522\BAR_SQLVIRT_DEV,

Line 2

Could not create constraint or index. See previous errors.

Solution:

There are two ways to get the missing column.

Method 1:

1. Open SQL Server Management Studio

2. Expand Security for the instance running HPSM database

3. Expand the Logins

4. Right-mouse click on the user running the SQL installation scripts and select Properties

5. Change the default Language to English -us_English as shown in the following screenshot:

103

6. Click on Ok

7. Close SQL Server Management Studio.

8. Now you can re-run the InstallSQLScripts with the account which was just changed.

Note: The InstallSQLScripts.zip are in the root directory of HPSM. Unpack the scripts, open a

command prompt with administrator rights and run InstallOrUpgradeRemoteDB.bat (required

syntax in corresponding help file and will be displayed when executing the bat file without any

further options.

9. Refresh the Devices screen in HPSM and the devices will now be displayed.

10. As an alternative to step 8, you can also reopen SQL Server Management Studio with the

following next steps:

• open the UpgradeDbSchemaVer21to22.sql from the InstallSQLScripts

• change on the first line:

USE $(DBNAME) into USE HPIPSC (or whatever the actual databasename is)

• Click on Execute

• Refresh the Devices screen in HPSM and the devices will now be displayed.

Method 2:

With this method the SQL user configuration remains the same.

1. Execute the following SQL commands in SQL Management Studio (in the example below

the database name is HPIPSC):

USE HPIPSC

'=============================================================================

PRINT 'Add CertExpiryDate Column in DeviceTable'

'=============================================================================

IF NOT EXISTS(SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME =

'DeviceTable' AND (COLUMN_NAME = 'CertExpiryDate'))

BEGIN

DECLARE @SQL NVARCHAR(MAX) = N'';

SET @SQL += N'

ALTER TABLE dbo.DeviceTable

ADD CertExpiryDate datetime NOT NULL DEFAULT CONVERT(DATETIME, ''9999-12-31

23:59:59.000'',120)'

PRINT @SQL

EXECUTE(@SQL)

END;

Note: 120 is used for the format ‘yyyy-mm-dd hh:mm:ss’

2. Refresh the Devices screen in HPSM and the devices will now be displayed.

When you click on Column Organizer, the column option Certificate Expiry Date should also be

displayed. If this is not displayed, remove all cached data in Chrome:

Go to Settings,

Privacy and Security

Clear Browsing data,

Select Time Range: All time

104

Select Browsing history, Cookies and other site data and Cached images and files.

Click on Clear Data

After re-logging into HPSM the column should now be displayed.

Certificate Installation Failures

Certificate installation fails on HP DesignJet Z9

HPSM can generate a CSR on behalf of the device (CSR source set to HPSM) or can retrieve the

CSR from the device, if the device supports this process. As this is not supported by the HP

DesignJet Z9, you almost must select the CSR source as HPSM for ID certificate installations on

this device.

In the EAPDeviceLib.log you can see the following error when using device as source:

2020-06-18 11:47:29,192 DEBUG Pipeline [5] - uid=_a738e1943df8_10.10.10.119, Exception while converting

certificate to X509Certificate

The input is not a valid Base-64 string as it contains a non-base 64 character, more than two

padding characters, or an illegal character among the padding characters.

Certificate installation issues with Microsoft CA

Troubleshooting certificate installation failures is no different than troubleshooting most

configuration issues. The typical scenarios can cause failures such as name resolution issues,

network connectivity issues, traffic blocked by firewall, permissions, device issues, etc.

Security Manager uses DCOM over RPC to submit requests to the CA and retrieve certificates,

like workstations do for auto-enrollment of certificates. Remote Procedure Call (RPC) is a

mechanism that allows Windows processes to communicate with one another, either between a

client and server across a network or within a single system. Numerous built-in Windows

components utilize RPC. RPC uses dynamic ports for communication between systems, but a

static port (TCP port 135) must also be used as a starting point for communication. The RPC

endpoint mapper listens on this static port.

In a typical RPC session, a client contacts a server's endpoint mapper on TCP port 135 and

requests the dynamic port number assigned to a particular service. The server responds with

the IP address and port number that the service registered with RPC when it started, and the

client then contacts the service on that IP address and port.

If the RPC server is unavailable, errors will occur indicating the certificate was not installed.

Many other reasons can cause a certificate to not install. For example, the RPC server's name

may be resolving to the wrong IP address, resulting in the client contacting the wrong server or

attempting to contact an IP address not currently in use. Alternatively, the server's name may

not be resolving at all. A firewall or other security application on the server, or a network firewall

appliance between the client and server, may be preventing traffic from reaching the server on

TCP port 135. The client may be unable to reach the server at all due to a general network

problem.

105

The following troubleshooting steps should help to resolve these issues.

1. Check the policy settings again for accuracy.

a. Check especially the Certificate Authority Server and Certificate Authority settings.

b. Ping the server by name from the client to verify that the name resolves to the

correct IP address.

c. If it does not, verify that the client and server are both using the correct DNS servers,

which must be inside the domain and will typically be domain controllers.

d. Try an IP Address instead of a hostname for the server in case the hostname is not

resolving.

e. Check whether the Key Length or Signature Algorithm values in the policy are not

supported by the device either as a value that can be created in a CSR if Jetdirect is

chosen as source or as a value in the certificate itself.

2. Check the Certificate Authority settings again to ensure that the account running the HPSM

service has the rights to submit requests to the CA.

NOTE: By default, the Network Service account runs the HPSM service, and Network Service

manifest itself remotely as the machine name (machine$).

3. Check the CA Template settings again.

a. Make syre the account running the HPSM service has the rights for Read and Enroll

and that Authenticated Users has Read permissions.

b. Make sure Submit in Request is selected in the template settings under the Subject

Name tab. If not, then certificate will be created for the Security Manager server

and not the printer.

4. Make sure the Security Manager server is on the same domain as the CA server.

NOTE: If the CA server is on a different domain as the HPSM server, and no trust relationship

exists between the domains, an error will display claiming the template does not exist. Even

though the template clearly exists, templates must be published into Active Directory for

clients to use them, and the lack of trust relationship will prevent Security Manager from

seeing the template.

5. Check firewall settings for any ports being blocked, and if the firewall is enabled on the

Security Manager server, make sure traffic on TCP port 135 is allowed to pass.

NOTE: Security Manager uses DCOM over RPC, like workstations do for auto-enrollment of

certificates, and DCOM uses port 135 for certificate enrollment. If workstations are

successfully auto-enrolling for certificates, it can be reasonably assumed the CA server

firewall is not blocking port 135.

The certutil tool can simulate the behavior Security Manager performs to submit a request and

retrieve a certificate by checking for the port being blocked or not:

certutil -ping “CA server”

106

Examples of an unsuccessful attempt to connect to a non-resolvable FQDN and a successful

attempt to the IP Address:

• The PortQry command-line utility or PortQryui.exe user interface utility (both

downloadable from Microsoft, for example: https://www.microsoft.com/en-

us/download/details.aspx?id=24009 ) can be used to test connectivity from the

client to the server and determine which ports are open on the server. It includes

support for RPC and can be used to determine which services have dynamic ports

registered with RPC and which specific ports they use.

• If workstations are also having issues auto-enrolling for certificates, then the

standard troubleshooting steps for resolving RPC Server Unavailable errors may

apply. For example, ensuring that the RPC service us running, the Authenticated

Users group is in the “Certificate Service DCOM Access” group, Enable Distributed

COM on this computer is selected in the Default Properties tab, etc.

107

Certificate installation issues with SCEP

Workflow for SCEP

When a SCEP server is used with a SCEP challenge password, then you should first see int the log

file, that the SCEP challenge password is requested (GetOneTimePassword) and obtained.

2021-04-09 16:26:43,506 DEBUG Scep [5] - Scep::GetOneTimePassword(): called

2021-04-09 16:26:43,568 DEBUG Scep [5] - Scep::SendRequest(): GET

https://Testserver.cloudshare.com/vedscep/mscep_admin

2021-04-09 16:26:46,814 DEBUG Scep [5] - Scep::SendRequest(): Server returned status code

OK (OK)

2021-04-09 16:26:46,830 DEBUG Scep [5] - Scep::SendRequest(): payload size: 2514

2021-04-09 16:26:46,830 DEBUG Scep [5] - Scep::SendRequest(): MIME content type: text/html

2021-04-09 16:26:46,830 DEBUG Scep [5] - --- BEGIN ndes_admin RESPONSE ---

__?<HTML><Head><Meta HTTP-Equiv="Content-Type" Content="text/html; charset=UTF-

8"><Title>Network Device Enrollment Service</Title></Head><Body BgColor=#FFFFFF><Font

ID=locPageFont Face="Arial"><Table Border=0 CellSpacing=0 CellPadding=4 Width=100%

BgColor=#008080><TR><TD><Font ID=locPageTitleFont Face="Arial" Size=-1

Color=#FFFFFF><LocID ID=locMSCertSrv>Network Device Enrollment

Service</LocID></Font></TD></TR></Table><P ID=locPageTitle> Network Device Enrollment

Service allows you to obtain certificates for routers or other network devices using the Simple

Certificate Enrollment Protocol (SCEP). </P><P> To complete certificate enrollment for your

network device you will need the following information: <P> The thumbprint (hash value) for the

CA certificate is: <B> 89AF30A1 77B4BA20 7B9E07FD F77643D9 </B> <P> The enrollment

challenge password is: <B> 365603DA91776E62 </B> <P> This password can be used only once

and will expire within 60 minutes. <P> Each enrollment requires a new challenge password. You

can refresh this web page to obtain a new challenge password. </P> <P ID=locPageDesc> For

more information see <A HREF=http://go.microsoft.com/fwlink/?LinkId=67852>Using Network

Device Enrollment Service </A>. </P> <P></Font></Body></HTML> __--- END ndes_admin

RESPONSE ---

2021-04-09 16:26:46,877 DEBUG Scep [5] - Scep::GetOneTimePassword(): Extracted ndes

challenge: 365603DA91776E62

Now HPSM can send the actual request:

2021-04-09 16:26:46,877 INFO Scep [5] - Generating CSR

2021-04-09 16:26:46,892 DEBUG Scep [5] - Scep::CreateSubjectName(): Created Subject Name

: CN = Printer1.usa.CompanyXXX, O = XXX, OU = MPS, L = Low, ST = New York, C = US

2021-04-09 16:26:47,455 INFO Scep [5] - Getting CA Certificates from SCEP Server

2021-04-09 16:26:47,455 DEBUG Scep [5] - Scep::GetCACert() called

2021-04-09 16:26:47,455 DEBUG Scep [5] - Scep::SendRequest(): GET

https://testserver.cloudshare.com/vedscep/mscep/?operation=GetCACert&message=ignore

2021-04-09 16:26:48,064 DEBUG Scep [5] - Scep::SendRequest(): Server returned status code

OK (OK)

2021-04-09 16:26:48,064 DEBUG Scep [5] - Scep::SendRequest(): payload size: 2378

2021-04-09 16:26:48,064 DEBUG Scep [5] - Scep::SendRequest(): MIME content type:

application/x-x509-ca-ra-cert

2021-04-09 16:26:48,080 DEBUG Scep [5] - Scep::GetCACert(): Received 2 CA/RA certificates

2021-04-09 16:26:48,095 INFO Scep [5] - Submit SCEP Request to SCEP Server

2021-04-09 16:26:48,095 DEBUG Scep [5] - Scep::SubmitToScep(): called

2021-04-09 16:26:48,264 DEBUG Scep [5] - Scep::CreateSubjectName(): Created Subject Name

: CN = printer1.usa.CompanyXXX, O = XXX, OU = MPS, L = Low, ST = New York, C = US

2021-04-09 16:26:48,264 DEBUG Scep [5] - Scep::SubmitToScep(): Selected SignatureAlgorithm

: SHA256

2021-04-09 16:26:48,592 DEBUG Scep [5] - Scep::SendRequest(): GET

https://uvo1pum39esurtb5qe9.env.cloudshare.com/vedscep/mscep/?operation=PKIOperation

108

&message=MIINNgYJKoZIhvcNAQcCoIINJzCCDSMCAQExCzAJBgUrDgMCGgUAMIIGtwY ---

removed further details for readability of log file—

2021-04-09 16:26:50,304 DEBUG Scep [5] - Scep::SendRequest(): Server returned status code

OK (OK)

2021-04-09 16:26:50,304 DEBUG Scep [5] - Scep::SendRequest(): payload size: 1202

2021-04-09 16:26:50,304 DEBUG Scep [5] - Scep::SendRequest(): MIME content type:

application/x-pki-message

2021-04-09 16:26:50,304 DEBUG Scep [5] - Scep::SubmitToScep(): Received PKCS7 Response

Packet

After this there is not much detail available about a potential error in the SCEP.log while

generating the certificate. In fact, the SCEP server only reports pkiStatus: 0 (successful),

PkiStatus: 3 (pending) or PkiStatus: 2 (failed) for the certificate generation.

Example of PKIStatus 3 in the HPCM.log:

2021-04-09 16:26:50,382 DEBUG Scep [5] - Scep::SubmitToScep(): pkiStatus:3 (Pending) failInfo:

(none)

This is followed by a polling request.

2021-04-09 16:26:50,382 INFO Scep [5] - Scep::SubmitToScep(): PKIStatus is3. Certificate Signing

Request pending. Poll to get the certificate

As polling is done by HPSM, it should be followed by a success message as well.

Example of success response 0 in the HPCM.log (can be seen with and without : )

2020-09-30 15:19:51,442 DEBUG Scep [4] - Scep::SubmitToScep(): pkiStatus:0 (Success) failInfo:

(none)

Example of failure 3 in HPCM.log:

2021-09-28 16:51:06,990 ERROR HPCM [28] - SCEPConnector: Errow while Enroll: PKIStatus is 2.

Error while signing the certifica

When an error 2 is reported, then additional tools will have to be used to find out why the process

failed on the SCEP server. Check the windows event viewer on the SCEP server, when this error

occurs. Also check if there are any logs available in the SCEP server, which might explain the

actual error message.

ASN1 bad tag in HPSM_service.log file using HPSM 3.7.0 or 3.7.1 and Microsoft

NDES (SCEP)

When using NDES (Microsoft implementation of SCEP), the ID certificate installation will fail with

HPSM 3.7 and 3.7.1 with the following error message in the HPSM_service.log file:

2022-03-23 14:52:23,477 ERROR Service [26] - PKI Provider Web Access Restriction Error.

Exception: HP.HPCM.Contract.Exception.BadRequestException: ASN1 bad tag value met.____ at

HP.HPCM.ScepConnector.SCEPConnector.Enroll(CertificateEnrollData certificateEnrollData)

at HP.HPCM.Provider.ProviderHandler.ExecuteEnroll(String providerName,

CertificateEnrollData certData)__ at HP.HPCM.HPCMService.Execute(CertificateEnrollData

request)

In the event viewer from the NDES server you might see the following error:

- <EventData Name="EVENT_MSCEP_FAIL_TO_DECRYPT_INNER">

109

<Data Name="ErrorMessage">BadData.</Data>

Solution

Upgrade to HPSM 3.8 or obtain new dll’s for HPSM 3.7.1 from HPSM support.

‘Bad Request to the Pki Provider’ or Cannot Remediate when using dynamic

passwords

Issue

When using Microsoft SCEP (NDES) ID, the certificate installation might fail. In the Assessment

report the following error will be listed:

Bad Request to the PKi Provider. Kindly check the UI/Credential details. Cannot Remediate.

Figure: “Bad Request to the PKi Provider….” Error listed

110

First check the credentials, SCEP URL and SCEP challenge password URL in the policy.

111

If the URLs and username and password are correct, then additional information should be

checked.

Check in the event viewer system log if there is an event ID 4.

Figure: Event Viewer, System log where an Event 4 is listed

Text from General section in EventID 4: The Kerberos client received a

KRB_AP_ERR_MODIFIED error from the server vm-w2k16$. The target name used was

HTTP/VM-W2K16.VM.ZBook. This indicates that the target server failed to decrypt the ticket

provided by the client. This can occur when the target server principal name (SPN) is

registered on an account other than the account the target service is using. Ensure that the

target SPN is only registered on the account used by the server.

This error can also happen if the target service account password is different than what is

configured on the Kerberos Key Distribution Center for that target service. Ensure that the

service on the server and the KDC are both configured to use the same password. If the

server name is not fully qualified, and the target domain (VM.ZBOOK) is different from the

client domain (VM.ZBOOK), check if there are identically named server accounts in these two

domains or use the fully qualified name to identify the server.

Here you already see that there is something wrong with the account (SPN) being used in the

policy.

In the HPSM Service Log you will find the same information as in the assessment report:

Line 563: 2022-04-05 15:04:51,246 ERROR Service [17] - PKI Provider Web Access Restriction

Error. Exception: HP.HPCM.Contract.Exception.BadRequestException: BadRequest__ at

HP.HPCM.ScepConnector.SCEPConnector.Enroll(CertificateEnrollData certificateEnrollData)

at HP.HPCM.Provider.ProviderHandler.ExecuteEnroll(String providerName,

CertificateEnrollData certData)__ at HP.HPCM.HPCMService.Execute(CertificateEnrollData

request)

112

In the SCEP Log, you will find some additional information to an unauthorized action:

2022-04-05 15:04:46,756 DEBUG Scep [4] - SCEP::SendRequest(): GET http://vm-

w2k16.vm.zbook/CertSrv/mscep_admin/

2022-04-05 15:04:47,055 ERROR Scep [4] - SCEP::SendRequest(): Exception

System.AggregateException: One or more errors occurred. --->

System.Net.Http.HttpRequestException: An error occurred while sending the request. --->

System.Net.WebException: The remote server returned an error: (401) Unauthorized. --->

System.ComponentModel.Win32Exception: The target principal name is incorrect__ at

System.Net.NTAuthentication.GetOutgoingBlob(Byte[] incomingBlob, Boolean throwOnError,

SecurityStatus& statusCode)__ at System.Net.NTAuthentication.GetOutgoingBlob(String

incomingBlob)__ at System.Net.NegotiateClient.DoAuthenticate(String challenge,

WebRequest webRequest, ICredentials credentials, Boolean preAuthenticate)__ at

System.Net.NegotiateClient.Authenticate(String challenge, WebRequest webRequest,

ICredentials credentials)__ at System.Net.AuthenticationManagerDefault.Authenticate(String

challenge, WebRequest request, ICredentials credentials)__ at

System.Net.AuthenticationState.AttemptAuthenticate(HttpWebRequest httpWebRequest,

ICredentials authInfo)__ at System.Net.HttpWebRequest.CheckResubmitForAuth()__ at

System.Net.HttpWebRequest.CheckResubmit(Exception& e, Boolean& disableUpload)__ ---

End of inner exception stack trace ---__ at

System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)__ at

System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)__ --- End of inner

exception stack trace ---__ --- End of inner exception stack trace ---__ at

System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)__ at

ScepClient.Scep.SendRequest(Uri requestUri, String method, Byte[] request_body, String

request_content_type, List`1 add_headers)__---> (Inner Exception #0)